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Preface 


This  report  documents  the  SCE  implementation  procedures 
developed  for  the  Electronic  Systems  Center  (ESC),  Hanscom 
Air  Force  Base.  The  information  contained  in  this  report  was 
developed  before  the  release  of  the  Software  Capability 
Evaluation  Version  2.0  Method  Description  [SCE  94]. 
Inconsistencies  should  be  checked  against  that  document 
which  takes  precedence  in  matters  of  SCE  methodology. 

Abstract  Software  Capability  Evaluation  (SCE)  offers  a  means  to 

evaluate  an  organization's  software  process  capability — that 
is,  how  well  an  organization  manages  the  process  it  uses  to 
create  software.  SCE  provides  a  way  to  compare  a 
development  organization's  software  process  against  a 
predefined  standard.  This  document  is  an  implementation 
guide:  it  is  intended  as  a  set  of  practical  information  which 
program  managers  can  use  to  guide  them  through  the  process 
of  using  SCE  in  an  acquisition. 


Purpose  of  This  The  Software  Engineering  Institute's  (SEI)  Software 
Document  Capability  Evaluation  Method,  as  introduced  in  this  guide, 

offers  a  means  to  help  acquisition  managers 

•  Identify  program  risk  by  evaluating  software  process 
capability  in  source  selection. 

•  Manage  program  risk  by  motivating  contractors  to 
improve  their  software  development  processes  without 
forcing  compliance  to  specific  practices. 

This  guide  can  be  used  to  implement  the  SCE  Method  in  order 
to  achieve  the  goals  above.  It  provides  specific  information 
necessary  to  orchestrate  SCE  use  during  the  source  selection 
process.  Specifically,  this  guide 
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•  Provides  guidance  on  how  to  use  the  SCE  method  as  a  tool 
to  identify  software  risk  during  a  source  selection. 

•  Provides  standardized  SCE  implementation  guidance 
which  is  documented,  available  for  review  and  comment, 
and  periodically  modified  as  experience  is  gained  with  its 
use. 

•  Provides  information  which  will  help  acquisition 
organizations  develop  appropriate  policies,  implementing 
instructions,  and  guidelines  to  use  SCE  in  source  selection 
and  institutionalize  SCE  as  a  routine  practice. 

•  Supplements,  but  does  not  replace,  team  training  for 
evaluating  the  software  process  capability  of  contractors. 


The  Software  Capability  Evaluation  Method  was  originally 
developed  by  Watts  Humphrey  and  William  Sweet  of  the  SEI, 
with  contributions  from  R.  K.  Edwards,  G.  R.  LaCroix,  M.  F. 
Owens  and  H.  P.  Schultz  of  the  MITRE  Corporation. 

This  document  was  written  by  a  team  consisting  of  Rick 
Barbour,  Paul  Bjrmes,  and  Bob  Lang. 

This  document  is  based  largely  on  the  previous  work  that  led 
to  the  SCE  Implementation  Guide  Version  1.0.  Thanks  are  again 
offered  to  all  of  the  folks  who  contributed  to  developing, 
writing  and  reviewing  that  document.  Many  reviewers  have 
contributed  to  the  development  and  revision  of  this 
document;  their  valuable  insights  added  a  great  deal  to  the 
quality  of  the  finished  product. 
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The  primary  audierxce  for  this  document  is  program  office 
personnel  responsible  for  the  software  component  of  an 
acquisition.  The  guide  assumes  that  the  program  manager 
will  be  delegated  the  responsibility  for  determining 
appropriate  SCE  usage,  developing  plans  for  implementing 
SCE,  and  carrying  out  the  SCE  plan.  Program  managers 
should  be  familiar  with  the  material  contained  in  Part  A  of  the 
guide,  as  a  minimum. 

Although  this  document  provides  examples  and  models  of 
SCE  use  in  source  selections  and  talks  to  some  specific  details 
of  source  selection,  it  is  a  guide  and  is  not  intended  to  replace 
the  use  of  organizational  regulations  on  source  selection.  In 
order  to  work  any  source  selection  successfully  it  is 
imperative  to  work  with  the  local  procurement  and  legal  staff 
as  well  as  the  source  selection  and  specific  acquisition 
regulations,  which  take  precedence  over  this  document. 

In  cases  where  SCE  is  being  used  in  a  source  selection,  source 
selection  personnel  should  be  familiar  with  Parts  A  and  B  of 
this  guide.  Part  A  provides  an  overview  of  the  SCE  Method 
and  its  uses.  Part  B  discusses  the  details  of  incorporating  SCE 
into  source  selections  of  specific  acquisitions. 

Part  A  of  this  guide  provides  an  introduction  to  SCE  that  all 
readers  should  understand.  It  provides  an  overview  of  SCE's 
technical  basis  along  with  a  high  level  view  of  the  mechanics 
of  executing  an  SCE  in  an  acquisition  and  a  discussion  of  the 
underlying  Capability  Maturity  Model  (CMM).  The  emphasis 
in  Part  A  is  on  providing  the  basic  imderstanding  of  the  SCE 
Method  necessary  for  a  user  to  focus  on  the  source  selection 
use  of  the  SCE  Method. 

Part  B  focuses  on  how  to  implement  SCE  in  a  source  selection. 
This  portion  of  the  guide  introduces  the  key  activities  and  is 
followed  by  specific  guidance  on  how  to  use  the  method  and 
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the  documentation  needed  to  execute  recommended 
approaches.  The  examples  included  are  modified  instances  of 
an  implementation  of  SCE.  There  may  be  other  approaches 
that  are  equally  valid  and  useful.  Some  audiences  may  only  be 
concerned  with  a  subset  of  the  information  about  the  SCE 
method  contained  in  this  guide;  consequently,  the  material 
has  been  organized  so  that  different  audiences  can  focus 
attention  on  specific  pieces. 
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Chapter  1  Introducing  SCE 


This  part  of  the  document  introduces  the  Software  Capability 
Evaluation  (SCE)  Method  and  relates  it  to  the  SEI  Capability 
Maturity  Model  (CMM).  Chapter  1  describes  what  SCE  is  and 
Chapter  2  discusses  the  execution  of  SCE  on  an  acquisition. 

Background  Early  attempts  at  imj^roving  the  acquisition  process  began 

with  a  memorandum  published  by  Deputy  Secretary  of 
Defense  Frank  C.  Carlucci  HI  in  1981.  Initiative  11  of  this 
memorandum  [Carlucci  81]  required  the  Department  of 
Defense  (DoD)  to  increase  the  visibility  of  technical  risk  in  the 
budgets  of  acquisition  programs  for  weapon  systems.  In  1986, 
the  United  States  General  Accounting  Office  (GAO)  released 
a  report  titled  "Technical  Risk  Assessment — ^The  Status  of 
Current  DoD  Efforts"  [GAO  86],  which  examined  the 
methodology  used  for  assessing  technical  risks  within  25 
program  offices.  The  deficiencies  found  by  GAO  prompted 
development  of  various  risk  assessment,  evaluation,  and 
management  publications,  including  the  Defense  Systems 
Management  College  (DSMC)  guide,  "Risk  Management 
Concepts  and  Guidance."  [Deep],  DoD  organizations 
launched  further  initiatives  to  improve  their  risk  assessment 
capability.  One  such  initiative  resulted  in  development  of  the 
SCE  Method. 

The  SCE  Method  complements  earlier  work  by  extending 
technical  risk  assessment  to  include  the  software  process.  It 
establishes  software  process  capability  as  a  criterion  for 
source  selection  by  providing  an  orderly  way  to  compare 
offerors'  software  capability  against  a  predefined  process 
maturity  model.  SCE  should  be  used  to  augment  other 
software  risk  assessment  techniques  currently  used  in  source 
selection  and  contract  monitoring. 
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Purpose  of  SCE 


The  SCE  Method  was  conceived  and  developed  by  the  SEI 
and  the  MITRE  Corporation  in  1987,  at  the  request  of  the  Air 
Force.  It  was  designed  to  help  program  managers  determine 
the  software  process  capability  of  a  contractor  at  or 
organizational  site  (facility  or  location). 

SCE  provides  a  snapshot  of  a  contractor's  past  process 
implementation,  current  process  activities,  and  future  process 
potential.  The  SCE  site  visit  is  an  in-plant  review  conducted 
by  four  to  six  persormel  over  a  three  day  period  at  a 
contractor's  facility.  The  output  from  an  SCE  site  visit  is  a  sc 
of  findings;  of  strengths,  weaknesses,  and  improvement 
activities  measured  against  the  CMM.  The  CMM  is  the 
technical  basis  for  reporting  the  findings  of  the  SCE  Method. 
The  CMM  is  described  in  the  following  two  documents: 

•  Capability  Maturity  Model  for  Software,  Version  1.1  [Paulk 
93a] 

•  Key  Practices  of  the  Capability  Maturity  Model,  Version  1.1 
[Paulk  93bl. 


The  SCE  team  identifies  an  offeror's  strengths,  weaknesses, 
and  any  improvement  activities  in  relation  to  the  CMM  and 
the  sponsoring  organization's  objectives.  The  findings  of  the 
SCE  team  are  incorporated  into  the  source  selection 
sponsoring  organization's  technical/management  team  for 
incorporation  into  acquisition  decisions.  The  SCE  team 
assimilates  data  on  various  project  practices  and  from  these 
creates  an  overall  picture  of  the  organization's  software 
process  capability  relative  to  the  CMM. 

Cost,  schedule,  and  performance  are  high  priorities  for  the 
program  manager.  A  basic  tenet  of  the  SCE  Method  and  CMM 
is  that  the  more  mature  the  contractor's  software  processes 
are,  the  more  likely  it  is  that  the  contractor  can  meet  predicted 
cost,  schedule,  and  performance  targets.  Because  SCE 
evaluates  an  organization's  software  process,  acquisition 
organizations  gain  insight  into  this  key  area,  an  area 
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The  Capability 
Maturity  Model 

Basic  Concepts 


The  Maturity  Model 
and  Software 
Quality 
Improvement 


traditionally  not  evaluated  in  the  past.  Note  that  SCE 
supplements  the  use  of  other  considerations  such  as 
application  domain  expertise,  past  performance,  and 
organizational  capacity  in  acquisition  decisions. 


The  CMM  is  based  on  the  premise  that  the  quality  of  a 
product  stems  in  large  part  from  the  quality  of  the  process 
used  to  create  it.  To  improve  product  quality  in  the  Total 
Quality  Management  (TQM)  sense,  the  process  used  for 
developing  the  products  should  be  defined,  understood, 
measured,  and  progressively  improved.  As  process  quality 
increases,  management  also  has  greater  insight, 
understanding,  and  control  of  risks.  Concepts  of  process  and 
quality  management  are  applied  to  building  and  maintaining 
software  products  by  using  the  CMM  in  conjimction  with  the 
SCE  Method. 

Capability  Maturity  Model  for  Software  [Paulk  93a]  describes  the 
framework  of  the  CMM  (vl.l).  The  CMM  organizes  common, 
proven  software  development  practices  into  a  structured 
framework  that  can  be  used  to  focus  quality  improvement 
efforts.  Teams  use  the  SCE  Method  v2.0  to  evaluate 
contractors  against  the  CMM  vl.l  (as  of  October  1993). 
Previously,  SCE  teams  were  trained  using  the  maturity 
framework  as  described  in  Characterizing  the  Software  Process: 
A  Maturity  Framework  [Humphrey  87a]  and  A  Method  for 
Assessing  the  Software  Engineering  Capability  of  Contractors 
[Hvunphrey  87b],  referred  to  in  this  document  as  CMM 
version  0. 

The  maturity  model  discussed  in  Capability  Maturity  Model  For 
Software  [Paulk  93]  consists  of  five  maturity  levels  with  key 
process  areas  (KPAs)  assigned  to  each.  Figure  1-1  shows  the 
name  and  number  of  each  level  in  the  left  colunrn.  Level  one 
is  the  lowest;  five  is  the  highest.  The  general  characteristics  of 
an  organization  functioning  at  a  particular  maturity  level  are 
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listed  in  the  middle  column,  and  the  KPAs  associated  with 
each  level  of  the  maturity  model  are  on  the  right.  This  maturity 
model  is  the  foundation  upon  which  the  CMM  was  built.  A 
reading  of  Key  Practices  of  the  Cajjability  Maturity  Model  [Paulk 
93b]  will  reveal  an  expansion  of  the  CMM  into  more  practical 
aspects  of  software  engineering. 


Lavel 

Characterietic 

Key  Process  Area 

Optimizing  (5) 

•  Continuous  process 
improvement  capability 

•  Process  change  management 

•  Technology  innovation 

•  Defect  prevention 

Managed  (4) 

•  Product  quality  planning 
and  tracking  of 
measured  software 
process 

•  Software  Quality  Management 

•  Quantitative  Process 
Management 

Defined  (3) 

•  Life  cycle  process 
defined  and 
institutionalized  to 
provide  product  quality 
control 

•  Peer  Reviews 

•  Intergroup  coordination 

•  Software  product  engineering 

•  Integrated  software 
management 

•  Training  program 

•  Organization  process 
definition 

•  Organization  pr(H.ess  focus 

Repeatable  (2) 

•  Management  oversight 
and  tracking  of  project 

•  Stable  planning 

•  Software  Configuration 
Management 

•  Softwar>9  quality  assurance 

•  Software  subcontract 
management 

•  Software  project  tracking  and 
oversight 

•  Software  project  planning 

•  Requirements  management 

InHial  (1) 

•  Ad  hoc  (unpredictable, 
chaotic) 

Figure  1-1  Capability  Maturity  Model  Version  1.1 


The  KPAs  are  "stepping  stones"  for  moving  to  higher  levels — 
that  is,  a  company  must  be  proficient  in  the  KPAs  within  a 
maturity  level  in  order  to  move  up  to  the  next  level.  For 
instance,  without  a  stable  and  repeatable  software  project 
planning  system,  investments  in  a  formal  definition  of  the 
organization's  technical  software  process  aren't  likely  to 
overcome  the  limitations  imposed  by  poor  software  project 
planning.  The  model  is  organized  with  basic  project 
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management  practices  at  the  lowest  levels,  and  the  more 
sophisticated  practices  supporting  product  development  at 
the  higher  levels.  While  moving  up  to  higher  levels,  a 
company  must  improve  as  well  as  maintain  proficiency  in  the 
KPAs  of  the  lower  levels. 

The  model  is  structured  to  demonstrate  that  the  greatest 
benefit  from  quantitative  measures  of  the  process  come  when 
all  of  the  KPAs  through  the  defined  level  are  successfully 
implemented  first.  The  CMM  in  effect  provides  a  step  ladder 
that  management  can  use  to  prioritize  scarce  resources 
towards  the  greatest  long  term  benefit  to  the  organization. 
With  this  structure  it  is  predicted  that  an  organization  can 
better  imderstand  the  processes  implemented  throughout  the 
company  and  design  an  improvement  plan  with  better 
chances  of  success.  Consequently,  tiiis  understanding  leads  to 
more  efficient  and  effective  investments  in  people,  process, 
and  technology  which  are  the  key  elements  of  a  sound 
organizational  improvement  program. 


Five  basic  levels  of  process  maturity  have  been  defined  in  the 
model  to  describe  the  progression  from  an  ad  hoc  software 
process  to  one  that  is  under  statistical  control  and  can  act  as  a 
foundation  for  continuous  process  improvement. 

Level  1  (initial):  Projects  can  be  characterized  as  routinely 
being  late  and  exceeding  the  planned  budget. 

Level  2  (repeatable):  The  organization  installs  basic 
management  controls  and  generally  learns  to  manage  its  costs 
and  schedules  while  building  similar  software  products.  The 
focus  is  on  the  product,  and  the  management  system  is  largely 
reactive. 

Level  3  (defined):  The  process  is  understood  and  explicitly 
defined.  Here,  the  organization  introduces  a  structured 
framework  for  software  development  and  establishes 
dedicated  process  improvement  resources. 


CMU/SEI-94-TR-5 


1-5 


Chapter  1:  Introducing  SCE 


The  Model-Based 
Structure  of  the 
CMM 


Level  4  (managed):  The  processes  are  quantified,  measured, 
and  reasonably  well  controlled.  Data  is  available  to  establish 
improvement  priorities  and  to  support  tool  and  technology 
investment.  In  statistical  process  control  terms,  special  causes 
of  process  variation  are  under  control. 

Level  5  (optimizing):  The  common  causes  behind  process 
variations  are  systematically  addressed,  and  process  data  is 
used  to  improve  the  process  in  response  to  new  and  evolving 
issues  and  capabilities.  The  organization  focuses  on 
continuous  improvement  guided  by  process  data. 


Figure  1-2  shows  how  tiie  various  aspects  of  the  CMM  relate 
to  one  another.  Maturity  levels  are  the  highest  abstraction, 
while  the  key  practices  are  the  most  detailed  portion  of  the 
model.  The  key  indicators,  and  hence  the  maturity 
questionnaire,  are  derived  from  the  key  practices. 
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Inherent  ability  of 
next  project  to 
meet  quality, 
schedule,  and  cost 
goals 


Ability  of  a  process 
to  control  or  avoid 
significant  causes 
of  poor  quality, 
cost,  and  schedule 
performance 


Organization 
process  maturity 
required  for  next 
maturity  level 


The  software 
processes  that 
are  main 
contributors  to 
maturity  level 
process  capability 

Principal  software 
practices  for 
achieving 
effective, 
reproducible 
process  results 


The  practices, 
expressed  as 
questions,  that 
have  the  highest 
reliability  in 
indicating  that  a 
process  area  is 
satisfied 


Effects  of  Increasing 
Maturity  On 
Program 
Predictability 


Figure  1-2  Capability  Maturity  Model  Structure 

Another  important  premise  of  the  CMM  is  that  increasing 
process  maturity  leads  to  reduced  variance  in  the 
performance  of  the  software  process  and  increased  software 
process  capability,  as  illustrated  in  Figure  1-3.  Organizations 
at  the  initial  maturity  level  will  miss  the  target  performance 
on  most  of  their  projects.  Occasionally,  a  strong  manager  may 
drive  one  project  in  a  Level  1  organization  to  a  higher  level  of 
process  capability,  but  that  capability  is  not  present 
throughout  the  organization.  As  organizations  reach  higher 


CMU/SEI-94-TR-5 


1-7 


Chapter  1 :  Introducing  SCE 


maturity  levels,  there  is  a  more  effective  infrastructure  in 
place  to  drive  the  process  capability  applied  on  the  next 
acquisition  to  achieve  program  goals. 


With  increasing  maturity: 

•  accuracy  increases 

•  variance  reduces 

•  target  improves 


Level  3 


Level  4 


Level  2 


Figure  1-3  Predicted  Distribution  of  Performance  as 
Organizational  Process  Maturity  increases 

The  defined  level  is  the  starting  point  from  which  to  achieve 
statistical  process  control,  which  enables  an  organization  to 
imderstand  where  and  how  the  quality  and  productivity  of  a 
process  can  be  continuously  improved.  Level  5  (optimizing)  is 
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the  level  at  which  data  is  available  to  tune  the  process  itself. 
The  optimizing  level  is  not  intended  to  be  an  end-point, 
however,  and  should  properly  be  viewed  as  a  beginning  of 
orderly  and  managed  process  improvement.  The 
evolutionary  journey  from  Level  1  to  Level  5  can  then  be  seen 
as  building  the  orgaiuzational  capability  to  make  sustained 
and  continuous  improvement. 


The  SCE  team  determines  what  process  activities  are  actually 
implemented  within  a  development  organization.  Figure  1-4 
depicts  the  separate  software  process  areas  defined  in  the 
CMM.  The  shaded  boxes  reflect  areas  covered  by  the  CMM 
key  process  areas  (KPAs). 
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Figure  1-4  Level  2  and  3  Key  Process  Areas  in  the  CMM 

Figure  1-4  illustrates  a  key  difference  between  traditional 
views  of  software  development  process  and  the  CMM.  The 
activities  within  the  dashed  rectangle  are  essentially  product 
building  activities  (e.g.  Requirements  Analysis,  Design, 
Testing).  Traditionally  software  product  development  has 
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focused  on  product  building  activities  such  as  these.  But  this 
traditional  approach  ignores  many  key  process  activities 
(shown  as  shaded  boxes)  which  are  crucial  to  successful 
product  building. 

The  SCE  team  examines  not  only  the  part  of  the  software 
development  process  that  t5rpically  shows  up  in  the  Software 
Development  Plan  (dashed  rectangle  depicted  as  software 
product  development  activities),  but  also  the  forces  acting  on 
the  process  and  the  relationships  between  the  forces. 
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Oriented 

Perspectives 


Figure  1-5  shows  the  DoD-STD-2167A  software  development 
activities  associated  with  the  typical  Engineering 
Manufacturing  Development  (EMD)  program.  Most 
programs  throughout  the  system  acquisition  lifecycle  are 
managed  by  a  lifecycle  development  model  that  is  composed 
of  these  activities. 


System 

Req. 

Analysis/ 

Design 


Software 

Req. 

Analysis 


SRR 

SDR 


Preliminary 

Design 


SRS 


Detailed 

Design 


PDR 


Coding  and 
CSU 
Testing 


CSC 

Integration 

Testing 


CDR 


CSCI 

Testing 


System 
Integration 
and  Test 


TRR 


SRR  •  System  Requirements  Review 

SDR  -  System  Design  Review 

SRS  •  Software  Requirement  Specification 

PDR  -  Preliminary  Design  Review 

CSU  -  Computer  Software  Unit 

CSC  -  Computer  Software  Component 


CSCI  •  Computer  Software  Configuration  Item 
CDR  -  Critical  Design  Review 
TRR  -  Test  Readiness  Review 
FCA  -  Functional  Configuration  Audit 
PCA  -  Physical  Configuration  Audit 


FCA/ 

PCA 


Figure  1-5  DoD-STD-2167A  Activities 

Each  of  these  development  steps  are  made  more  effective  by 
support  from  the  KPAs  in  the  CMM.  Historically, 
government  programs  have  focused  on  the  development  of 
software  products  without  regarding  the  supporting 
processes  as  equally  important.  DoD-STD-2167A  reflects  this 
bias  toward  product  delivery.  The  KPAs  provide  a 
supporting  process  environment  in  which  the  organization 
can  make  effective  decisions  regarding  the  deliverables 
shown  in  Figure  1-5.  Thus  a  focus  on  KPAs  balances  the 
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historical  product  orientation  with  a  process  focus  that  is 
critical  in  predicting  contractor  performance  and  measuring 
product  development  expertise. 


Operational 
needr  — 


System  Requirements 
Process 


SSDD 

product 


Interface  Requirements 
Process 
IRS  product 


SSDD 

product 


Software  Requirements 
Process 


SRS  products 


1  per  CSCI 


Software  Requirements 
Engineering  and 
Preliminary  Design 

Process 


Detailed 

Software  Design 
Process 


SDD  detailed  _ 

design  level  (CSUs) 


System/Segment 
Design  Document  (SSDD) 


Software  Requirements 
Specification  (SRS) 

Spec,  of  required  software 
capabilities 


Z 

SDD  product 

Software  Test  Plan 

preliminary 

DI-MCCR-80014A 

design 

r 

Software  Design 
Document  (SDD) 
Spec,  of  software's 
preliminary  design 


Specification  of 
software's  detailed  design 


The  sum  of  the  activities  to 
product  deliverables  does  not] 
equal  the  process 


Figure  1-6  A  Product  Approach  to  Software  Process 

Figure  1-6  graphically  depicts  a  product  documentation 
approach  to  software  process  often  adopted  by  acquisition 
and  contractor  management.  One  common  problem  is  that 
people  often  equate  the  organization's  software  process  to  the 
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creation  of  output  products  during  each  of  the  steps  shown 
above.  The  actual  software  development  process 
implemented  in  an  organization  contains  many  more 
activities  than  the  steps  directly  related  to  the  product 
building  parts  of  the  process.  Dociunents  defining  specific 
intermediate  products  are  not  the  process,  but  are  in  fact 
artifacts  of  the  process  which  is  implemented  in  an 
organization.  A  successful  program  manager  will  focus  on 
process  (as  well  as  product)  as  a  predictor  of  future 
performance  of  the  development  of  the  product  he  or  she  is 
tasked  to  acquire. 

Another  problem  with  the  product  view  of  software  process 
shown  in  Figure  1-6  is  the  fact  that  products  built  by  a 
software  development  organization  are  often  overcome  by 
technological  or  environmental  obsolescence  very  quickly.  In 
this  acquisition  environment,  it  is  difficult  to  estimate 
accurately  new  program  risks  by  investigating  existing 
systems  because  the  existing  systems  may  be  obsolete. 

New  program  risk  assessment  is  difficult  because  one  cannot 
assume  an  organization  has  the  ability  to  take  on  a  more 
technologically  advanced  or  larger  project,  make  effective  use 
of  new  technology,  or  address  changes  in  the  threat  or 
development  enviroiunent  based  on  past  project  success 
alone.  This  is  because  software  system  requirements  are 
rarely  the  same  from  project  to  project.  New  requirements 
and  advances  in  technology  inevitably  mean  that  old  ways  of 
conducting  business  will  be  inadequate.  Thus  it  is  essential  for 
organizations  to  focus  on  the  processes  that  support  the 
product  development  activities. 

Using  prior  data  to  evaluate  parameters  of  the  new  system 
assumes  the  new  system  is  "precedented,"  meaning  it  is 
similar  to  systems  previously  built.  According  to  the  Air  Force 
Studies  Board  [AFSB  89],  a  precedented  system  meets  three 
criteria: 

1 .  A  stable  set  of  software  requirements  exists  that  is  not 
substantially  different  from  that  of  a  previous  system. 
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2.  The  digital  system  architecture  and  software  design  that 
will  satisfy  the  given  requirements  are  known. 

3.  The  contractor's  system  engineering  and  software  teams 
communicate  effectively  with  each  other  and  have  prior 
experience  in  developing  a  similar  system. 

According  to  the  AFSB,  the  majority  of  USAF  systems  (and  by 
implication  all  modem  complex  weapon  systems)  can  be 
considered  imprecedented  in  that  major  portions  of  the 
software  development  do  not  meet  at  least  one  of  the  criteria 
above. 

The  ability  to  address  these  issues  is  as  much  dependent  on 
the  organization's  software  process  capability  as  it  is  on  past 
product  building  successes.  Risk  abatement  can  be  better 
achieved  by  evaluating  software  process  capability  in 
addition  to  using  traditional  product-specific  methods. 
Institutionalization  of  the  key  software  practices  provides 
another  indicator  of  how  the  organization  is  likely  to  address 
technological  challenges  and  manage  risks. 

A  program  manager  must  wrestle  with  product-oriented 
questions,  but  answers  to  these  questions  will  not  address 
many  of  the  software  process-specific  risks  in  acquisitions. 
Typical  capacity  reviews  derive  current  capability  from  past 
performance,  and  are  used  to  answer  many  product-oriented 
questions.  The  SCE  Method  adds  value  to  the  typical  capacity 
review  because  it  measures  a  subset  of  software  process 
attributes  that  are  believed  to  indicate  organizational  ability 
to  successfully  develop  the  product  to  be  acquired — in  other 
words,  to  take  on  future  challenges,  rather  than  past  efforts. 

An  example  which  illustrates  the  difference  between  a 
capacity  review  and  an  SCE  is  in  the  training  area.  Assume  a 
new  procurement  calls  for  500  KSLOC  in  Ada,  and  50  Ada 
programmers  are  expected  over  the  life  of  the  EMD  phase.  In 
a  capacity  review,  the  government  team  is  primarily 
concerned  with  whether  the  contractor  has  enough 
experienced  Ada  people  on  hand  to  perform  the  work.  In  an 
SCE,  the  emphasis  is  not  on  specific  people,  but  on  the  process 
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the  contractor  uses  for  training — in  this  case,  the  process  and 
plans  for  training  the  Ada  programmers  if  not  enough  are 
available  to  work  on  the  project. 

Both  process  and  product  perspectives  are  important  to  the  program 
manager.  The  contractor's  software  process  is  by  no  means  a 
program  manager's  only  concern.  But  if  the  program  manager 
focuses  on  process  capability  in  addition  to  traditional 
product-oriented  concerns,  risk  areas  can  be  identified  up 
front,  and  quality  problems  which  may  occur  on  die  program 
due  to  those  risks  can  be  proactively  managed  and  prevented. 


The  findings  from  an  SCE — strengths,  weaknesses  and 
improvement  activities  relative  to  the  CMM — can  influence 
the  source  selection  decision  or  contribute  to  the  future 
direction  of  an  ongoing  program.  However,  there  are  many 
other  attributes  besides  the  software  process  that  are 
important  in  determining  the  most  qualified  contractor  to  do 
a  job. 
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Contractor 

•  Manufacturing 

•  Tools  and  Instrumentation 

Attributes  That 

•  Con^xiter  Security 

•  Pattern 

Could  Be 

•  Si^ud 

Recognition 

Considered  in 

Processing 

•  Hardware 

a  Software-Intensive 

*  OiHsct-Aleftted 

Engineering 

Acquisition 

Programmbrg 

•  Maintainability 

•  System  En^ieering 

•  Ada  Expertise 

•  Communications 

•  Prior  Performance 

•  Software  Safety 

•  People 

Analysis 

•  Site  Capacity 

•  Formal  Methods 

•  Environments 

•  Software  Testing 

•  Location 

•  Human 

•  ReliabUity 

Englneedng 

•  Hardware 

•  Facflities 

Engineering 

•  Reuse 

•  Networks 

•  Requirements 

•  Integrated  Software 

Management 

Management 

•  Software  Project  Planning 

•  Software  Product 

•  Software  Project 

Engineering 

Tracking  and  Oversight 

•  Intergroup  Coordination 

•  Software  Subcontract 

•  Peer  Reviews 

Management 

•  Software  Quality 

Key  Process  Areas 

^  •  Software  Quality 

Management 

Considered  by  SCE 

1^  Assurance 

•  Quantitative  Process 

•  Software  Configuration 

Management 

Management 

•  Defect  Prevention 

•  Organization  Process 

•  Technology  Innovation 

Focus 

•  Process  Change 

•  Organization  Process 
Definition 

Management 

Figure  1-7  Sample  of  Contractor  Attributes  That  Couid  Be 
Considered  in  a  Software-Intensive  Acquisition 

Figure  1-7  depicts  the  software  intensive  attributes  of  a 
contractor's  organizational  site  and  those  considered  by  SCE. 
an  SCE  site  visit  to  a  contractor's  facility  is  an  intense 
examination  of  the  orgaiuzation  that  reveals  its  software 
development  process  capability  and  improvement  activities. 
All  of  the  attributes  shown  above  are  important  to  an 
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acquisition,  but  only  those  related  to  software  process  are 
captured  and  recorded  in  the  findings  during  the  SCE  site 
visit. 

Areas  other  than  software  process — tools,  systems 
engineering,  and  skill  and  experience  with  a  particular 
language — should  be  investigated  also.  These  items  should  be 
investigated  outside  the  structure  of  the  SCE  site  visit  to 
ensure  repeatability,  consistency,  and  reliability  of  the 
method.  This  is  important  to  ensure  fairness  and  equitable 
treatment  of  all  competing  offerors  during  a  source  selection. 
While  the  SEI  supports  the  need  to  review  other  critical  areas 
that  are  not  covered  in  the  CMM  KPAs,  no  attempt  to  merge 
these  areas  into  the  SCE  findings  should  be  made  on  site.  The 
current  method  of  teaching  and  conducting  an  SCE  site  visit 
follows  the  decomposition  of  the  CMM  along  the  lines  of  the 
KPAs. 

Findings  of  strengths,  weaknesses,  and  improvement 
activities  against  the  CMM  KPAs  are  the  result  of  an  SCE.  The 
SCE  Method  is  essentially  a  data  collection  activity  which 
extends  insight  into  an  area  which  has  been  lacking  in  the 
past:  software  process  capability.  Figme  1-8  shows  a  sample 
of  a  detailed  finding  for  the  Software  Project  Planning  KPA. 
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Software  Project  Planning 

Commitment  process  at  senior  and  first-line  management  levels 
requires  strengthening 

Strengths 

•  Project  responsibilities  are  defined  and  documented 

•  Mechanism  in  place  to  assure  that  software  estimates  are  tracked 
in  relation  to  software  activities 

Weaknesses 

•  Commitment  procedures  at  senior  and  first-line  management 
levels  could  not  be  validated  by  the  team 

Improvement  Activtties 

•  Task  group  is  in  place  to  address  senior  management  issue 


Figure  1-8  Sample  Key  Process  Area  Finding 

The  SEI  has  developed  Software  Process  Assessments  (SPA) 
and  SCE  as  part  of  a  strategy  to  improve  the  state  of  the 
practice  in  software  engineering.  Both  methods  address  the 
SEI  vision  of  supporting  the  evolution  of  software 
engineering  from  an  ad  hoc,  labor-intensive  activity  to  a 
managed,  technology-supported  engineering  discipline. 

Differences  in  the  methods  stem  from  the  motivations, 
objectives,  outcomes,  and  ownership  of  findings.  These 
factors  lead  to  substantive  differences  in  interview  dynamics, 
scope  of  inquiry,  information  being  gathered,  and 
formulation  of  the  outcome  (findings). 
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Figure  1-9  highlights  several  of  the  major  differences  between 
the  two  methods  which  affect  the  way  they  are  performed. 


AssessiiMnt 

Evaluation 

Used  by  organization  to  improve  its 
software  processes 

Used  by  acquisition  organization  in 
source  selection  and  contract 
monitoring 

Assesses  process  practice 

Substantiates  current  practice 
process 

Acts  as  catalyst  for  process 
improvement 

Evaluates  contractor’s  commitment  to 
improve 

Provides  input  for  improvement 
action  plan 

Analyzes  contract  performance 
potential 

Findings  may  include  issues  not 
explicit  in  the  Capability  Maturity 

Model 

Rndings  restricted  to  Capability 

Maturity  Model  issues 

Collaborative:  members  of 
organization  must  be  on  team 

Third  party  oriented:  members  of 
organization  not  on  team 

Applies  to  organization,  not  individual 
projects,  contracts 

Organizational  data  but  applied  to  a 
specific  contract  award  on  acquisition 

Input  for  improvement  action  plan  to 
unfreeze  organization 

Input  for  award  decision,  contract 
monitoring,  or  risk  management 

Figure  1-9  Differences  Between  Process  Assessments  and 
Capability  Evaluations 

Critical  differences  an  SCE  user  must  understand  are  the  basic 
assumptions  built  into  the  methods  themselves.  First,  the 
current  SPA  method  assiunes  that  members  of  the 
organization  will  cooperate  openly,  fully,  and  in  a  manner 
that  provides  factual  contributions.  This  asstimption  is  made 
because  presumably  participants  have  no  reason  to  mislead 
SPA  team  members,  who  are  from  their  own  organization, 
and  are  dedicated  to  improving  the  organization  with  the  full 
support  and  commitment  from  senior  management.  The  SPA 
findings  are  intended  to  be  incorporated  into  action  plans  for 
orgaruzational  improvement,  and  therefore  do  not  verify 
findings,  by  examination  of  documentation,  every  assertion 
made  during  the  interview  process.  SCE  assumes  that 
members  of  the  evaluated  organization  will  make  every 
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attempt  to  put  their  organization's  software  process 
capability  in  the  best  possible  light.  This  is  because  their 
company's  livelihood  and  therefore  their  own  careers  and 
livelihoods  may  be  at  stake  since  the  SCE  findings  are  used  as 
factors  in  determining  potential  monetary  awards  to  the 
company.  Thus,  every  attempt  is  made  by  the  SCE  team  to 
verify  facts  through  use  of  a  document  review  process. 

an  SCE  team  consists  of  one  acquisition  team  leader  and  three 
to  five  acquisition  personnel  and/or  their  engineering 
support  contractor  personnel.  This  team  may  include  a 
procurement  member  or  observer.  The  SPA  team  consists  of 
six  to  eight  members  either  entirely  from  the  organization 
itself,  or  from  both  the  organization  and  either  the  SEI  or  one 
of  the  SETs  licensed  SPA  vendors. 

Finally,  SPAs  are  longer  in  duration,  involve  more  people 
from  the  assessed  organization,  and  are  generally  more  costly 
than  evaluations.  SPA  data  is  not  recommended  for  use  on 
government  source  selections  because  of  the  inherent 
differences  between  the  two  methods  as  explained  above. 
Little  information  from  SPAs  can  be  directly  applied  by  an 
acquiring  organization  for  the  purpose  of  selecting  the  most 
appropriate  offeror. 

The  methods  are  similar  in  that  they  both  use  the  framework 
of  the  CMM  to  structure  a  detailed  investigation  into  software 
practices,  and  both  require  extensive  training  and  experience 
to  conduct  properly.  SPA  training  does  not  prepare  a  team  to 
perform  SCEs,  and  SCE  training  does  not  prepare  a  team  to 
conduct  SPAs. 

Both  the  SPA  and  SCE  methods  seek  to  identify  the 
organization's  software  process  strengths  and  weaknesses  as 
measured  against  the  KPA  goals  of  the  CMM.  Both  seek  to 
motivate  the  contractor  to  address  the  major  weaknesses  in  an 
aggressive  software  process  improvement  plan.  However,  the 
source  of  the  improvement  motivation  is  clearly  different. 
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Facilitating  SCE 
on  Your 
Program 


The  findings  from  an  SCE  deal  strictly  with  organizational 
strengths  and  weaknesses  against  the  CMM,  not 
recommendations  to  rectify  the  problems  as  is  the  case  with 
SPA. 

The  differences  between  the  two  methods  are  important  to 
acquisition  managers.  It  is  important  because  the  acquisition 
organization  may  benefit  in  the  long  run  by  conducting 
business  with  contractors  that  are  implementing  software 
process  improvements.  Although  SCE  use  may  focus  the 
attention  of  senior  executives  on  investments  in  software 
process  improvement,  the  greatest  benefits  will  be  apparent 
to  those  firms  who  have  embraced  the  concepts  of  process 
improvement  without  acquisition  organization  pressures  or 
incentives.  Acquisition  managers  must  understand  the 
lunitations  posed  by  SCE  and  SPA,  and  how  they  relate  to 
their  acquisition. 


Integrating  the  SCE  Method  into  an  acquisition  involves  four 
steps: 

1.  Identifying  the  maturity  of  the  contractor's  current 
software  process  in  terms  of  strengths,  weaknesses,  and 
any  improvement  activities  relative  to  the  CMM  KPAs. 

2.  Assessing  program  risk  and  how  the  contractor's 
improvement  plai\s  alleviate  that  risk. 

3.  Making  continuous  process  improvement  a  part  of  the 
contractual  relationship  with  the  contractor. 

4.  Monitoring  software  process  performance  and  developing 
a  working  relationship  with  the  contractor  (as  opposed  to 
an  adversarial  relationship). 

The  SEI  provides  briefings  and  presentations  to  acquaint 
sponsoring  organizations  with  the  evaluation  method.  If  a 
sponsoring  organization  decides  to  use  the  evaluation 
method,  it  sends  representatives  to  the  SEI  training  courses: 
SCE  Overview  Seminar  and  SCE  Team  Training.  The 
overview  helps  the  sponsoring  organization  realize  the 
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implications  and  benefits  of  using  software  capability 
evaluations,  and  the  team  training  helps  teams  prepare  to 
conduct  an  SCE.  They  can  then  conduct  a  pilot  use  of  SCE. 
After  completing  several  pilots,  the  sponsoring  organization 
decides  whether  to  iitstall  the  evaluation  method  on  a  broader 
scale.  A  typiccil  timeline  for  transferring  the  evaluation 
method  into  routine  practice  is  outlined  in  Figure  1-10. 
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- 

Contact 

Presentations,  briefings,  papers 

Awareness 

SEI  Course:  SCE  Ovenriew 

Executive  commitment  and  iunding 
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- 

Plan  for  pilot  use  (Selected  programs) 

SCE  team  selection 
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SEI  Course:  Evaluation  Team  Training 
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Plan  for  installing  SCE 

Tailor  SCE  products 
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24  - 
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Draft  and  improve  policy  on  SCE  use 

Revise  and  approve  policy  on  SCE  use 

Develop  an  institutional  SCE  capability 

30  - 

- 

Institution¬ 

alization 

SEI  Course:  Train  the  SCE  trainers 

SCE  use  is  organizational  practice 

Figure  1-10  Steps  for  Transferring  the  Evaiuation  Method 
into  an  Organization 


Once  the  CMM  is  imderstood,  the  different  applications  of  the 
CMM  are  recognized,  a  sponsoring  organization  can  begin,  at 
a  macro  level,  completing  the  activities  of  the  checklist  foimd 
in  Figure  1-11. 
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Planning  Tha  Imptemantation  of  Softarare  Capability 
Evaluation 


□  Attend  SEI  Course;  SCE  Overview 

□  Selected  engineering  staff  should  read  remainder  of  guide 

□  Determine  program  use 

□  Include  SCE  Method  in  appropriate  documents 

□  Select,  register,  and  train  SCE  team 

□  Conduct  SCE,  save  results,  and  capture  lessons  learned 

□  Brief  the  SEI  on  use  of  SCE 


Figure  1>1 1  SCE  Implementation  Checklist 

The  SCE  Method  is  flexible  because  the  application  of  the 
method  can  be  tailored  without  changing  the  conduct  of  the 
site  visit.  Thus,  different  types  of  acquisitions,  types  of 
processes,  size  of  an  organization,  and  degrees  of  process 
automation  or  use  of  took  can  be  accommodated  by  the 
method.  It  k  acceptable  to  have  alternative  approaches  to 
implementing  SCE  within  a  specific  context.  The  intent, 
however,  is  to  have  the  site  visit  itself  be  identical,  a  "black 
box,"  from  site  to  site.  In  other  words.  Army  Material 
Command  (AMC)  may  do  source  selections  differently  from 
Air  Force  Electronic  Systems  Center  (ESC)  and  the  Naval 
Command,  Control  and  Ocean  Surveillan::e  Center 
(NCCOSC),  RDT&E  CHvkion  (NRAD),  but  all  should  perform 
the  SCE  Method  the  same  way.  Thk  ensures  flexibility  in  the 
use  of  the  method  while  ensuring  reliability,  consktency,  and 
repeatability  of  the  evaluation  method  itself. 

Below  are  important  roles  when  SCE  k  being  applied  in  a 
government  acquisition: 

The  Source  Selection  Authority  (SSA),  who  k  responsible  for 
the  efficient  and  proper  conduct  of  the  source  selection 
process. 

The  Source  Selection  Advisory  Council  (SSAC),  which  ranks 
offeror  proposak  according  to  an  evaluation  plan. 
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The  Source  Selection  Evaluation  Board  (SSEB),  which 
evaluates  offeror  proposals. 

The  Source  Selection  Evaluation  Team  (SSET)  is  a  combined 
SSEB  and  SSAC.  They  perform  the  responsibilities  set  forth  in 
APR  70-15  and  APR  70-30.  ESC  uses  either  an  SSEB  or  SSET  on 
their  source  selections,  from  here  on  the  use  of  SSEB/T  or 
SSEB  will  refer  to  either  SSEB  or  SSET,  whichever  structure  is 
being  used. 

The  Procuring  Contracting  Officer  (PCO),  who  is  responsible 
for  solicitations  and  contracts,  communications  with  offerors, 
consistency  of  the  source  selection  plan  with  the  Pederal 
Acquisition  Regulation  (PAR),  contract  award,  and  other 
requirements  and  functions  specified  in  the  FAR  except  the 
source  selection  responsibilities  of  the  SSA. 

The  Program  Manager  (PM),  who  is  responsible  for 
developing  and  implementing  the  acquisition  strategy, 
preparing  the  source  selection  plan,  and  obtaining  SSA 
approval  of  the  plan  before  issuance  of  the  Request  For 
Proposal. 

The  Software  Capability  Evaluation  team  is  sponsored  by 
the  acquisition  organization  and  consists  of  a  group  of  four  to 
six  appropriately  experienced  (e.g.  education,  domain 
expertise,  numbers  of  years  experience)  persoimel  who  are 
trained  in  applying  the  SCE  Method.  The  SCE  team  is 
responsible  for  applying  the  SCE  Method,  including 
preparing  for  and  conducting  the  site  visits  and  reporting  the 
findings. 


Care  in  implementing  SCE  is  important.  The  SCE  team  must 
be  properly  selected  and  trained  to  prepare  the  team  for  the 
site  visit.  Only  experienced  and  trained  teams  can  use  the  SCE 
Method  in  the  intended  manner.  Training  consists  of  two 
courses.  The  first.  Software  Capability  Evaluation  Overview 
Seminar,  provides  a  briefing  on  the  concepts,  benefits,  and 
guidelines  and  logistics  of  SCE  to  the  acquisition  management 
team.  The  second,  SCE  Team  Training,  provides  a  review  of 
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software  process  capability  relative  to  the  CMM,  how  to 
conduct  an  SCE  site  visit,  and  team  development  activities 
including  case  studies  for  SCE  team  members. 
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Some  of  the  steps  required  to  prepare  for  an  SCE  are  specific 
to  its  use.  Preparation  for  performing  an  SCE  site  visit 
includes  the  following: 

•  Determining  SCE  use. 

•  Selecting  the  SCE  team. 

•  Training  the  SCE  team. 

•  Providing  directions  to  the  contractor. 

•  Screening  contractor  responses, 

•  Preparing  on-site  materials. 

•  Coordinating  on-site  activities. 

The  timeline  in  Figure  2-1  shows  when  each  activity  should 
occur.  It  is  generic,  and  is  not  reflective  of  actual  effort 
required  for  each  task.  Each  program  office  should  tailor  this 
schedule  of  activities  to  meet  the  needs  and  constraints  of  its 
specific  acquisition. 

Determining  Note  in  Figure  2-1  the  significant  amovmt  of  time  allowed  for 

SCE  Use  determining  how  to  use  SCE.  Organizations  should  use  this 

time  to  consider  the  varying  methods  of  implementation  and 
the  individual  techniques  and  procedures  that  may  be  used. 
The  range  of  time  for  each  activity  will  vary  with  the 
experience  of  the  team.  Those  organizations  familiar  with  SCE 
use  will  spend  significantly  less  time  on  the  intervening  steps 
once  the  decision  to  use  SCE  has  been  made. 
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The  acquisition  organization  must  decide  how  to  factor  the 
SCE  findings  into  the  acquisition.  Part  B  of  this  document 
explores  these  issues  and  others  a  program  office  must 
consider  to  use  SCE  in  a  source  selection.  Experience  has 
demonstrated  the  need  for  preparing  as  early  as  possible, 
particularly  in  large,  understaffed  acquisition  offices. 
Additional  preparation  time  may  need  to  be  factored  into  the 
schedule  in  anticipation  of  gaining  approval  to  proceed  with 
the  SCE.  For  example,  contracts  for  small  or  small, 
disadvantaged  businesses  require  extra  preparation  and 
approvals  outside  the  normal  chain  of  command. 


Selecting  the  Selecting  qualified,  experienced  software  acquisition 

SCE  Team  persormel  to  serve  as  SCE  team  members  is  one  of  ffie  most 

difficult  and  important  tasks  a  program  or  software  manager 
may  do  in  the  course  of  implementing  SCE  in  an  acquisition. 
The  key  considerations  for  assembling  a  SCE  team  are 

•  Individual  experience. 

•  Team  skills  and  e?q>erience. 

•  Individual  interpersonal  skills. 
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Individual 

Experience 


Team  Skills  and 
Experience 


•  Team  leadership  skills,  team  building  activities,  and  team 
staffing. 

SCE  team  members  should  be  selected  from  the  most 
experienced  people  in  the  program  office,  qualified  personnel 
from  field  organizations  (laboratories),  qualified  engineering 
support  contractor  (e.g.,  MITRE,  Aerospace,  IDA),  and 
people  from  other  program  offices  that  can  be  used  on  a 
temporary  basis.  The  most  successful  teams  will  be  those  that 
average  at  least  seven  years  of  software  and/or  software 
acquisition  experience  across  the  team.  At  least  one  member 
of  the  team  should  have  source  selection  experience.  This  is 
important  because  what  can  and  cannot  be  done  during  a 
source  selection  is  different  from  what  is  permissible  after 
award.  An  acceptable  spread  of  experience  levels  that  have 
been  found  to  be  successful  in  an  SCE  team  is 

•  At  least  one  or  two  senior  personnel  (more  than  seven 
years  appropriate  experience). 

•  Two  or  three  mid-level  personnel  (five  to  seven  years 
appropriate  experience). 

•  One  junior  person  (two  to  four  years  appropriate 
experience)  Note:  This  is  a  recommended  maximum. 
Junior  personnel  typically  will  not  have  the  background  to 
vmderstand  certain  aspects  of  software  processes  they  will 
observe. 

The  background  of  SCE  team  members  should  strike  a 
balance  between  software  technical,  software  management, 
and  software  acquisition  experience.  They  should,  as  a 
minimum,  share  a  mix  of  knowledge  and  experience  in  the 
following  software  engineering  disciplines: 

•  Acquisition  policies  and  procedures  (particularly  source 
selection  procedures). 

•  Project  management  and  planning. 

•  Software  configuration  management. 
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Interpersonal  Skills 


Team  Leadership 
Skills 


•  Software  design^  development  and  methodologies. 

•  Software  quality  assurance. 

•  Systems  engineering. 

•  Technical  requirements  of  the  contract. 

•  Testing. 

•  Application  domain,  e.g.  Avionics,  Missiles,  C3I, 
databases. 

In  general,  expertise  will  be  necessary  from  at  least  one  team 
member  in  each  of  the  key  process  areas  to  be  investigated. 
Certain  areas  may  be  stressed  over  others  depending  on  the 
acquisition. 

SCE  team  members  must  be  "team  players."  Conducting 
SCEs  requires  professional  judgement  and  team  consensus — 
attributes  that  are  necessary  to  work  effectively  in  an  SCE 
team  are  patience  and  perseverance.  Past  experience  has 
demonstrated  that  if  team  members  lack  interpersonal 
skills — essential  to  fostering  good,  open  communications 
between  team  members  and  the  contractors — ^the  team  is  less 
effective,  less  credible,  and  less  motivated  to  fulfill  the  SCE 
objectives.  The  ability  to  communicate  with  the  contractor 
and  other  team  members  is  the  essence  of  SCE  teamwork. 

Experience  shows  that  an  effective  team  leader  is  critical  to  the 
operation  of  the  team.  The  team  leader 

•  Ensmes  that  the  team  meets  its  schedule  and  objectives, 
encourages  teamwork  and  consensus  building. 

•  Is  the  SCE  team  focal  point  for  both  the  acquisition  office 
and  the  contractors  (planning,  scheduling, 
commxmicating). 

•  Should  have  enough  basic  leadership  skills  to  ensure  that 
the  team  functions  effectively. 
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The  team  leader  should  be  the  one  most  qualified,  based  on 
knowledge,  experience,  and  amoimt  of  direct  SCE  site  visit 
experience.  Occasionally,  teams  can  break  down  when  an 
inappropriate  team  leader  is  appointed.  Program  office 
management  should  prevent  this  from  happening.  Previous 
SCE  experience  should  be  a  criterion  for  assignment  as  an  SCE 
tecun  leader.  New  SCE  team  leaders  should  have  participated 
on  at  least  two  SCEs  as  a  team  member  before  assuming  a 
leadership  role.  This  experience  of  participating  on  SCEs  will 
prepare  the  new  leader  to  understand  the  SCE  team  process, 
team  dynamics,  and  contractor  sensitivities. 

Team  Building  An  essential  aspect  of  preparing  a  team  to  conduct  an  SCE  is 

Activities  performing  team  building  activities  before  going  on  site. 

Assume  the  SCE  team  has  never  worked  together.  An  activity 
that  would  help  bring  the  individual  members  together  as  a 
team  could  be  an  exercise  whereby  a  simple  task  is  assigned 
and  discussed.  For  example,  each  team  member  would 
interview  the  person  to  his  or  her  right  at  a  table  or  in  a  room. 
The  task  of  the  interviewer  is  to  leam  the  person's 
backgroimd,  interests,  and  area  of  expertise.  Each  team 
member  would  then  introduce  and  briefly  state  the  results  of 
their  interview.  The  team  could  then  identify  its  relative 
background  expertise  areas  to  the  evaluation  task  they  are 
being  asked  to  perform.  For  reference,  the  bibliography  in 
Appendix  C  of  this  guide  contains  three  references,  [Bucholz 
87],  [Denton  89],  [Kelly  91],  that  contain  more  on  teams  and 
team  building  activities.  These  exercises  will  help  determine 
how  the  team  members  work  together.  Often,  many  months 
may  pass  after  teams  have  completed  SCE  team  training  and 
before  they  conduct  site  visits.  The  team  building  activities 
are  important  for  the  team  members  to  re-acquaint 
themselves  as  a  single  operating  entity  able  to  reach 
consensus  effectively.  There  may  be  times  when  trained 
individuals  are  brought  together  who  have  not  completed 
training  together.  In  this  scenario,  team  building  is  crucial, 
since  they  have  not  operated  as  a  team  before. 


CMU/SEI-94-TR-5 


2-31 


Chapter  2:  Preparing  to  Use  SCE 


Team  Staffing 


Training  the  SCE 
Team 


The  success  of  the  SCE  team  hinges  on  its  ability  to  identify 
and  reach  consensus  on  the  strengths  and  weaknesses  of  a 
contractor.  An  SCE  team  is  neither  an  autocracy,  where  the 
leader  dictates  what  decisions  are  made,  nor  a  democracy, 
where  the  team  votes  and  the  majority  prevaik.  Instead  the 
decisions  are  reached  by  team  consensus,  meaning  all 
members  agree  to  the  findings,  and  there  is  no  significant 
minority  dissent  on  issues.  If  consensus  on  an  issue  carmot  be 
reached,  then  there  is  no  finding  in  that  area.  This  is  where 
team  building  activities  will  pay  large  dividends. 

Staffing  the  team  is  another  issue  for  consideration.  Creating 
a  "software  center  of  excellence"  is  an  excellent  mechanism 
for  building  a  core  of  individuals  who  are  highly  skilled  in 
conducting  SCE. 

System  Program  Offices  will  normally  draw  SCE-trained 
persormel  from  within  their  own  organization.  If  this  pool 
does  not  have  enough  individuals,  a  request  to  the  "center  of 
excellence"  organization  would  then  be  made  to  assist  in 
identifying  team  members.  In  this  manner,  the  program  office 
can  take  advantage  of  key  components  mentioned  above 
imder  individual  and  team  skills.  This  alternative  makes 
better  use  of  limited  software  skilled  resources  while  ensuring 
that  the  program  office  acquisition  expertise,  knowledge  of 
the  product  t3q)e  and  contractor  base,  and  "domain" 
knowledge  of  the  product  to  be  acquired  is  present  on  the 
team.  Furthermore,  the  core  resources  will  become  valuable 
assets  to  the  organization  as  they  gain  more  experience 
conducting  evaluations  for  multiple  acquisitions  in  different 
programs. 

Training,  preparation,  and  experience  separates  good  SCE 
teams  from  poor  ones.  There  are  three  levels  of  training 
needed  before  an  individual  should  be  considered  fully 
qualified  to  conduct  SCEs: 

•  Preparation 

•  Coursework 

•  Experience 
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Preparation 


Coursework 


The  preparatory  stage  consists  of  on-the-job  experience,  prior 
training,  and  professional  reading.  It  is  recommended  that 
SCE  team  members  take  courses  in  team  work,  assertiveness 
training,  and  total  quality  management.  Watts  Humphrey's 
book.  Managing  the  Software  Process  [Humphrey  89]  and  the 
latest  release  of  the  CMM  [Paulk  93a]  are  two  essential 
readings  that  are  provided  to  participants  of  the  evaluation 
team  training  course.  Participation  in  the  one-day  SCE 
Overview  Seminar  is  also  beneficial  background  to  prepare 
team  members. 

SCE  team  training  consists  of  a  multi-day,  case-study- 
oriented  course  that  all  SCE  team  members  must  successfully 
complete.  This  course  is  intended  for  experienced  personnel 
who  have  been  selected  to  conduct  the  SCE  Method.  It 
provides  the  knowledge  and  reinforces  the  skills  required  to 
conduct  SCEs  effectively,  and  helps  the  group  develop  into  a 
cohesive  team.  A  sampling  of  course  content  follows: 

•  Team  building  exercises. 

•  Preparing  for  the  site  visit. 

•  Conducting  interviews. 

•  Substantiating  key  practices. 

•  Reviewing  doaunents. 

•  Developing  and  presenting  findings. 

SCE  teams  need  effective  communicators  willing  to  take  on 
different  roles  (e.g.  facilitator,  recorder,  interviewer, 
timekeeper),  as  demanded  by  the  dynamics  of  the  team  and 
constraints  of  the  acquisition.  The  SCE  team  needs  to  know 
how  to  evaluate  the  contractor  in  relation  to  the  CMM,  so  a 
working  imderstanding  of  the  CMM  is  required.  Teams  are 
taught  that  processes  implemented  are  to  a  large  degree 
dependent  on  several  non-process  variables.  It  takes 
experience  to  understand  these  relationships  and  exercise 
professional  judgement,  which  is  why  the  team  characteristics 
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Experience 


and  profile  are  crucial  in  addition  to  the  coursework.  Roles 
taken  on  by  team  members  to  accomplish  the  site  visit  include 
the  following: 

•  Team  Leader:  manages  process,  keeps  to  agenda, 
guarantees  deadlines  and  deliverables. 

•  Facilitator:  sustains  team  spirit,  moves  team  toward 
consensus,  and  encourages  participation. 

•  Recorder:  captures  information,  does  not  editorialize,  and 
lists  documents  to  be  reviewed. 

•  Participant:  assists  other  team  members  and  carries  out 
assigned  tasks. 

Every  graduate  of  tire  SCE  training  program  should  be  a 
member  rather  than  a  leader  of  an  SCE  team  for  at  least  two 
SCEs.  Junior-  and  mid-level  p)ersonnel  should  take  part  in  at 
least  three  SCEs  before  being  considered  for  the  team 
leadership  role.  Resource  demands  and  time  constraints, 
however,  may  prevent  this  from  happeiung.  When  working 
vmder  such  constraints,  it  is  recommended  that  the  program 
office  send  the  team  to  practice  an  SCE  on  a  volunteered 
organizational  office  before  beginning  the  source  selection. 
One  acquisition  team  practiced  doing  SCEs  on  at  least  three 
occasions  to  insure  personnel  were  highly  trained  for  the 
source  selection. 

The  common  denominators  in  discussions  with  individuals 
returning  from  their  first  SCE  is  their  desire  for  more  team 
training,  preparation,  and  time  to  conduct  the  interviews. 
SCE's  activities  are  not  radically  different  from  those  done  in 
the  program  office  on  a  day  to  day  basis.  Taken  together, 
however,  they  are  group  activities  requiring  significant 
practice  and  preparation.  Practicing  as  a  group  will  reveal 
individuals'  strengths  and  weaknesses,  depth  of  required 
preparation,  and  how  to  manage  the  SCE  process  to  capitalize 
on  the  team's  strengths  so  that  more  effective  and  timely  SCEs 
are  conducted. 
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SCE  in  source  selection  provides  the  program  office  with 
several  opporturuti^  to  interact  with  the  potential  offeror(s). 
The  program  office  will  need  to  provide  directions  in  the 
Instructions  For  Proposal  Preparation  (IFPP)  explaining  how 
to  complete  the  preparatory  materials  that  form  the  initial 
baseline  of  the  site  visit.  They  will  also  have  to  present  the  SCE 
process  and  describe  how  the  findings  are  factored  into  the 
acquisition  at  either  a  Pre-proposal  Conference  or,  in  the 
acquisition  documents  or  both.  Part  B  of  this  document  will 
discuss  in  detail  the  areas  requiring  acquisition  organizational 
direction  and  interaction  with  the  contractor. 

For  this  discussion  the  use  of  SCE  will  affect  the  following 
acquisition-related  documents.  Documents  with  an  asterisk 
(*)  after  them  provide  direction  and  information  to  die 
contractor  community  in  an  acquisition,  while  the  others  are 
acquisition  organization  internal  documents. 

•  Commerce  Business  Daily  Announcement* 

•  Source  Selection  Plan  (SSP) 

•  Source  Selection  Evaluation  Guide  (SSEG) 

•  Pre-Proposal  Conference  * 

•  Request  for  Proposal  (RFP)*  -  Sections  H,  M,  L,  (BFPP) 

•  Possibly — ^Statement  of  Work  (SOW)  or  Award  Fee  Plan* 

•  Debriefings  of  Unsuccessful  Offerors* 


The  acquisition  organization  will  request  the  offerors  to 
prepare  and  submit  profiles  and  Maturity  Questionnaire 
responses  for  each  of  six  to  eight  projects  fiiat  are 
representative  of  the  software  development  work  at  the  site(s) 
that  is  being  proposed.  This  is  discussed  further  in  Part  B  of 
this  guide.  For  the  remainder  of  this  chapter  discussion,  only 
the  highlights  of  preparation,  conducting  the  site  visit  and  the 
final  reports  will  be  covered. 


CMU/S0-94-TR-5 


2-35 


Chapter  2:  Preparing  to  Use  SCE 


Requests  for  Other 
Information 
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Site  Visit 


Preparing  for  the 
Site  Visit 


The  contractor's  organization  charts  specific  to  the 
organizational  site  and  the  projects  to  be  submitted  for 
consideration  by  an  SCE  team  are  particularly  helpful  in 
building  a  preliminary  plan  of  who  is  to  be  interviewed  to 
discuss  certain  issues.  Request  after  competitive  range 
determination  and  before  site  visits  only  that  documentation 
which  the  SCE  team  has  time  scheduled  to  review  and  has 
factored  into  its  planning  activities.  Providing  documentation 
is  a  burdensome  tcisk  for  the  contractors  and  reviewing  it  is  a 
time-consuming  activity  for  the  SCE  team.  Select  only  what  is 
essential  based  on  analysis  of  contractors'  responses  and  only 
if  time  is  available.  Documents  that  will  reveal  processes  at 
work  should  be  selected  over  those  that  are  technical  in  nature 
or  discuss  plans.  (Plans  often  are  not  implemented  or  updated 
and  are  good  only  as  a  point  of  departure.) 


The  following  discussion  is  a  brief  overview  of  the  essential 
steps  required  to  execute  the  SCE  on-site  period.  It  is  included 
here  for  continuity  of  the  Part  A  Discussion.  Detailed 
discussion  are  available  in  a  separate  SEI  Document,  the 
Software  Capability  Evaluation  Version  2.0  Method  Description 
[SCE  93]. 

•  Developing  the  acquisition  product  profile  and  Target 
Process  Capability.  The  Target  Process  Capability  is  the 
explicit  identification  of  the  software  process  scope  to  be 
evaluated.  SCE  uses  the  KPAs  of  the  CMM  to  "target"  the 
areas  of  software  process  capability  investigation. 

•  Selecting  contractor  projects. 

•  Identifying  Critical  Subprocesses  for  all  contractors. 

•  Analyzing  the  contractor's  project  profiles  and  Maturity 
Questionneiire  responses  with  respect  to  the  product 
profile  and  the  TPC.  Note:  The  Maturity  Questionnaire  is 
used  only  as  an  input,  in  conjimction  with  the  analysis  of 
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Project  Profiles,  with  respect  to  the  Product  Profile  and  the 
Target  Process  Capability.  The  Maturity  Questionnaire  is 
not  scored  by  the  SCE  team. 

•  Determining  key  issues  for  individual  contractors. 

•  Developing  initial  exploratory  interview  questions. 

•  Developing  an  agenda  and  schedule  for  initial  exploratory 
interviews  and  document  reviews 

•  Notifying  individual  contractor  focal  points  of  SCE  team 
logistic  requirements. 

•  Presenting  the  arrival  brief  to  contractor  management. 

•  Analyzing  organizational  and  project  documentation. 

•  Reviewing  agenda  and  schedule. 

•  Conducting  exploratory  interviews. 

•  Requesting  additional  documentation. 

•  Validating  interview  respoi\ses. 

•  Drafting  preliminary  findings. 

•  Validating  preliminary  findings. 

•  Conducting  coitsolidation  interviews. 

•  Validating  improvement  activities. 

•  Collecting  final  data. 

•  Developing  final  findings. 

• 

Contracting  Officer  (PCO)). 

Using  its  collective  professional  judgement  and  a  consensus 
decision  making  process,  the  SCE  team  puts  together  its 
findings  from  individual  projects  to  create  a  set  of  overall 
findings.  These  findings  are  the  embodiment  of  all  the 
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interview  and  documentation  notes  developed  before  and 
during  the  on-site  review.  Detailed  findings  should  be 
prepared  for  each  KPA  investigated. 

Findings  should  be  specific  to  the  point  where  they  identify 
the  cause  for  a  strength  or  weakness,  but  not  so  specific  that 
the  finding  places  the  team  in  a  comer  by  failing  to  consider 
exceptions  that  may  exist  within  the  organization.  SCE  teams 
must  remember  they  aie  evaluating  a  subset  of  the  total 
projects  ongoing  at  a  site  as  a  proxy  for  predicting  the 
orgaiuzation's  capability  to  do  a  specific  project,  and 
exceptions  may  exist  because  of  this  process.  The  team  should 
be  prepared  to  substantiate  the  strengths  and  weaknesses 
without  attributing  the  information  to  specific  individuals  or 
projects.  Individual  confidentiality  is  a  vital  component  of  a 
good  site  visit.  The  team  should  create  and  maintain  a 
detailed  record  of  the  site  visit  which  the  contracting  officer 
can  include  as  part  of  the  documentation  making  up  the 
permanent  contract  file. 
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Part  B  Using  SCE  in  a  Source  Seiection 

Introduction  The  next  three  chapters  explore  in  depth  the  role  of  SCE 

throughout  the  source  selection  process.  First  a  high  level  look 
is  provided,  then  some  key  issues  which  should  be  considered 
when  using  SCE  are  explored.  Part  B  will  give  the  reader  a 
realistic  understanding  of  the  extensive  planning  required  to 
implement  SCE  in  a  source  selection.  SCE  is  more  than  a  series 
of  site  visits  followed  by  an  outbrief  to  the  Source  Selection 
Advisory  Council  (SSAC),  Source  Selection  Evaluation  Board 
(SSEB)  or  equivalent  organization  under  streamlined  source 
selection  procedures. 

Figure  B-1  provides  a  high-level  flow  chart  of  the  normal 
source  selection  activities  that  will  be  affected  by  the 
incorporation  of  SCE  into  tlie  soui  ce  selection  process.  The 
chart  simplifies  somewhat  the  multitude  of  interactions  that 
go  on  during  the  planning  and  execution  of  SCE.  The  first  step 
in  the  process  is  to  decide  whether  SCE  should  be  used  in  the 
source  selection  process. 

Chapter  3  places  SCE  in  the  context  of  a  typical  source 
selection  by  discussing  issues  that  should  be  considered. 
These  issues  include  criteria,  costs  and  benefits  of  using  SCE, 
both  for  the  acquisition  organization  and  for  the  offerors. 
Chapter  3  also  looks  at  the  personnel  involved  in  the  SCE 
process. 
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Figure  B>1  Source  Selection  Activities  Affected  By  SCE 

Once  the  organization  has  decided  to  use  SCE  in  an 
acquisition,  the  proper  role  for  SCE  in  the  source  selection 
criteria  must  be  assessed.  This  decision  point  is  labeled  with  a 
"\"  in  Figure  B-1.  Should  SCE  be  factored  in  as  a  Performance 
Risk  Assessment  or  Evaluation  Criterion  or  both?  Chapter  4 
discusses  some  alternate  methods  of  using  SCE  findings  in  the 
source  selection  decision.  It  also  addresses  the  implications  of 
using  SCE  on  tlxe  source  selection  schedule. 

The  boxes  labeled  2  through  5  address  documentation 
typically  required  for  a  source  selection,  whether  SCE  is  used 
or  not:  Source  Selection  Plan  (SSP),  Pre-proposal  Conference 
presentation.  Evaluation  Standard,  and  the  Request  For 
Proposal  (RFP).  Each  piece  of  documentation  must  address 
SCE  to  a  certain  degree.  Chapter  5  explains  how  to  develop 
these  documents  to  accommodate  SCE. 
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Like  all  schedules,  timetables,  or  flow  charts  found  in  this 
guide.  Figure  B-1  is  only  a  representation  giving  the  user  of 
SCE  insight  into  what  is  required  to  implement  SCE 
effectively.  Variations  to  the  schedules  and  timetables  should 
be  expected.  For  example,  SCE  training  could  easily  be  carried 
out  before  the  Pre-proposal  Conference  and/or  the  writing  of 
the  Evaluation  Standard  in  order  to  accommodate  the  vmique 
demands  of  the  acquisition. 
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Chapter  3  Deciding  to  Use  SCE 


Criteria  for 
Using  SCE  in  a 
Source 
Seiection 


This  section  introduces  the  issues  that  must  be  considered 
when  contemplating  use  of  SCE  in  source  selection.  General 
familiarity  with  the  acquisition  organization  source  selection 
process  is  assumed  for  purposes  of  focusing  on  each  player's 
relationship  to  SCE.  Appendix  A:  SCE  Participants  in  Source 
Selection  provides  a  brief  disciission  of  each  primary  Source 
Selection  participant  that  is  affected  or  can  affect  the  use  of 
SCE. 

Clearly,  SCE  should  not  be  used  for  every  software 
acquisition.  Both  the  costs  and  benefits  of  using  SCE  as  well  as 
the  specific  nature  of  the  acquisition  should  be  considered 
when  making  this  decision.  These  costs  and  benefits  may 
indicate  that  other  approaches  are  necessary  for  very  small 
acquisitions  involving  software.  This  section  discusses 
criteria  related  to  the  nature  of  an  acquisition. 

There  is  no  mmimiun  number  of  lines  of  code  or  measure  of 
software  intensity  dictating  that  SCE  must  be  used  in  an 
acquisition.  Several  considerations  must  be  weighed  by  the 
program  manager  when  making  the  decision.  Each  of  the 
following  factors  are  important  considerations,  but  the 
program  manager  or  person  responsible  for  determining  SCE 
usage  for  an  acquisition  must  weigh  these  factors  in 
accordance  with  their  organization's  method  of  procuring 
systems.  These  are  general  guidelines  that  must  be  refined  to 
meet  the  context  of  the  organization: 

•  Criticality  of  an  acquisition  or  the  software  component. 

•  Total  dollar  value  of  the  acquisition  or  software 
component. 

•  Management  control  priority. 

•  Unprecedented  system  mission  needs. 
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•  Acquisition  life  cycle  phase. 

•  Length  of  acquisition  time  period. 

•  Software  size,  the  number  of  Computer  Software 
Components  (CSCs),  and  the  prime  contractor  - 
subcontractor  relationship. 

Figure  3-1  illustrates  the  relationship  of  each  of  these  factors 
as  a  general  guideline  for  determining  appropriateness  of  SCE 
usage.  Each  box  should  be  read  independently,  and  then 
combined  with  other  factors,  to  make  an  overall  judgement  on 
SCE  applicability. 
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Figure  3-1  SCE  Usage  Decision  Making  Criteria 


3-44 


CMU/SEI-94-TR-5 


Chapter  3.  Deciding  to  Use  SCE 


Criticafity  of  an 
Acquisition  or  the 
Software 
Component 


Total  Dollar  Value  of 
the  Acquisition  or 
Software 
Component 


Management 
Control  Priority 


The  criticality  of  an  acquisition  may  necessitate  SCE  use.  The 
SEI  recommends  that  any  government-defined  major 
program  use  SCE  as  an  integral  part  of  its  strategy  for 
producing  the  highest  quality  end  product  and  motivating 
government  contractors  to  focus  on  software  process 
improvement  as  a  means  to  effect  this  goal.  In  all  Mission 
Critical  Computer  Resource  (MCCR)  systems,  regardless  of 
total  dollar  amoimt,  software  size,  or  DoD  priority  ranking, 
SCE  use  should  be  strongly  considered.  MCCR,  and  software 
in  general,  are  critical  components  of  modem  weapon 
systems.  The  success  of  the  system  is  largely  dependent  upon 
software  precisely  performing  its  intended  function.  An 
example  of  a  small,  but  highly  critical  piece  of  software 
warranting  the  use  of  SCE  in  an  acquisition  would  be 
software  needed  to  control  the  hardware  for  an  access  control 
system  for  nuclear  weapons  or  other  munitions. 

Dollar  value  is  important  because  of  the  investment  in 
resources  and  time  necessary  to  implement  SCE  effectively. 
Use  of  SCE  should  be  considered  when  the  total  value  of  an 
acquisition  software  component  exceeds  $10  million.  On  any 
acquisition  in  which  total  cost  is  greater  than  $25  million,  SCE 
use  should  be  strongly  considered. 

Where  the  software  component  of  a  program  itself  exceeds  $5 
million  or  is  greater  than  30%  of  the  total  program  cost,  SCE 
should  be  used.  This  criterion  may  often  take  precedence  over 
the  $25  million  threshold  described  above.  On  the  other  hand, 
some  acquisitions  in  the  $10  to  $25  million  range  may  not 
warrant  the  use  of  SCE  because  of  program-unique 
circiunstances.  Perhaps  the  software  component  is  not 
mission  critical  and  is  less  than  10%  of  the  total  dollar  value. 
These  guidelines  are  not  absolute;  they  are  intended  as 
guidelines  to  aid  the  decision-making  process  and  the 
development  of  appropriate  policies  and  procedures. 

When  management  control  is  a  high-priority  concern,  SCE 
use  should  be  considered.  An  environment  under  effective 
management  control  will  be  more  able  to  produce  data  that  is 
useful  for  lessons  learned  which  can  be  incorporated  into  the 
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overall  systems  development  process.  These  lessons  help  the 
acquisition  organization  avoid  "re-inventing  the  wheel." 
Successful  management  control  also  facilitates  effective 
implementation  of  modem  methodologies,  tools,  and 
techniques. 


A  controlled  environment  is  essential  to  managing  contractor 
processes — processes  for  maintaining  the  development 
enviroiunent,  bringing  new  people  and  technology  into  the 
environment,  identifying  problems  early  in  the  contract,  and 
managing  requirements  changes.  A  controlled  environment 
enables  improved  risk  assessment  and  abatement. 


SCE  should  be  used  when  the  contractor  is  likely  to  develop 
software  implementations  that  are  imprecedented.  When  the 
mission  of  the  system  system/software  component, 
especially  the  role  played  by  the  software  component,  is 
known  and  defined  by  the  end  user,  and  portions  of  the 
system  will  be  unprecedented,  use  of  SCE  should  be  strongly 
considered.  SCE  helps  identify  program  risks  associated  with 
the  capability  of  contractors  to  succeed  in  producing  quality 
software  in  an  imprecedented  environment. 


Use  of  SCE  yeilds  information  about  an  organization's  ability 
to  manage  risks  inherent  in  unprecedented  software 
development,  as  well  as  their  ability  to  manage  tasks  which 
are  new,  but  are  similar  to  ones  they  have  successfully 
completed  previously. 


Unprecedented  systems  (i.e.,  those  solving  new  or  imique 
problems)  pose  special  problems  for  software  development 
organizations.  An  SCE  of  each  offeror  would  identify  whether 
the  requisite  controls  are  in  place  on  the  contractors'  existing 
programs  and  whether  they  will  be  easily  transferred  to  the 
new,  unprecedented  system. 

The  lifecycle  phase  of  an  acquisition  is  an  important  factor  in 
determining  SCE  usage.  The  SCE  Method  and  CMM  were 
originally  developed  in  response  to  DoD's  and  industry's 
recognized  problems  in  managing  the  development  of 
increasingly  complex  mission  critical,  software-intensive 
products  in  the  real-time,  embedded  domain.  Given  this 
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Length  of 
Acquisition  Time 
Period 


background,  SCE  fits  in  any  engineering  manufacturing 
development  (EMD)  program  within  this  domain,  since  EMD 
is  the  typical  phase  associated  with  major  new  software 
development.  The  SEI  recommends  that  any  EMD  program 
consider  SCE  use,  in  accordance  with  the  other  factors  listed 
here.  However,  SCE  use  is  not  limited  to  the  EMD  phase.  The 
SCE  Method  has  been  used  successfully  in  demonstration/ 
validation,  concept  exploration,  and  operational  readiness 
support  phases. 

The  SCE  Method  should  be  considered  in  any  procurement 
where  software  is  a  major  component  and  the  program 
duration  period  is  expected  to  be  greater  than  24  months.  This 
timeframe  is  recommended  because  of  the  amoimt  of 
resources  necessary  to  apply  SCE  effectively,  and  because  the 
typical  process  improvement  program  implemented  by  a 
contractor  requires  at  least  18-24  months  to  attain  and  sustain 
improvements  in  process  maturity.  Thus,  more  software 
development  time  is  necessary  to  see  improved  results 
directly  on  the  contract. 

SCE  should  also  be  used  when  the  program  office  expects 
significant  block  upgrades,  modifications,  or  follow-on 
programs  to  occur,  and  the  original  contractor  is  expected  to 
be  a  primary  offeror  or  likely  performer  of  the  nt  n  work. 
Often,  the  processes  put  in  place  by  the  contractor  at  the  start 
of  a  software  development  will  be  frozen,  meaning  that 
process  changes  will  be  limited  during  that  development 
period.  Software  upgrades  or  major  modifications  to  existing 
systems  are  good  times  to  unfreeze  the  current  software 
development  process  and  install  new,  improved  processes, 
methods,  and  technology.  Therefore,  using  SCE  during  the 
initial  software  development  and  the  subsequent 
improvement  programs  will  enable  any  improved  processes 
to  be  implemented  on  the  follow-on  developments. 

SCE  use  may  still  be  appropriate  even  if  neither  of  these 
criteria  are  met  if  government  Program  Executive  Officer 
(PEO)  center/ commander  or  activity  committee  is  attempting 
to  motivate  and  gain  improvements  in  a  particular  domain 
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area,  such  as  avionics  systems.  These  PEO  decisions  may 
entail  long-range  coitsiderations  which  go  beyond  the  current 
contract  award,  and  thus  SCE  use  may  be  appropriate  to  meet 
other  government  objectives. 

The  amount  of  software  to  be  developed  will  directly  affect 
relationship  to  the  number  of  CSCs  required  to  effectively 
partition  the  software  system  into  manageable  chunks  and 
the  likelihood  of  a  prime  contractor  performing  integration  of 
software  produced  by  several  subcontractors.  When  the 
estimated  size  of  the  software  component  exceeds  100 
thousand  source  lines  of  code  (KSLCXZ),  SCE  should  be  used. 
At  this  threshold,  the  complexity  of  the  software  development 
will  likely  cause  the  ability  to  manage  a  large  number  of  CSCs 
and  subcontractors  to  be  a  significant  concern  of  the  program 
manager.  Scaling  up  small  software  engineering  teams  to 
meet  the  challenges  of  a  large  development  creates  additional 
pressures  on  effective  software  development  processes. 

There  are  several  realted  considerations  that  should  also  be 
weighed  by  the  program  manager  when  determining 
whether  to  use  SCE: 

•  Software  size  between  25  and  200  KSLOC. 

•  Minimum  development  team  of  20  to  100  people  in  under 
a  year  with  several  years  of  support  and  enhancements. 

•  Software  embedded  on  multiple  platforms  in  different 
languages  for  a  real-time  application. 

•  Highly  specialized  technologies:  for  example,  radar  signal 
processing  on  a  unique  programmable  signal  processor  or 
image  processing  on  a  customized  parallel  processor. 

•  Software  pieces  to  be  subcontracted  to  geographically 
distant  locations. 

These  examples  highlight  different  managerial /technical 
capabilities  a  contractor  must  possess  depending  on  the  type, 
complexity,  and  size  of  the  software  and  the  nature  of  the 
delivery  schedule. 


Software  Size, 
Number  of 
Computer  Software 
Components,  and 
the  Prime 
Contractor- 
Subcontractor 
Relationship 
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Benefits  of 
Using  SCE  in  a 
Source 
Selection 


A  clear  understanding  of  acquisition-specific  circiunstances, 
rather  than  knowledge  of  hard  criteria,  is  necessary  to 
determine  whether  SCE  use  is  appropriate.  In  general,  for  all 
acquisitions  with  a  software  component,  the  acquisition 
organization  should  seek  to  do  business  with  contractors  who 
understand  and  effectively  implement  modem  software 
development  practices,  and  also  with  those  contractors  taking 
actions  to  improve  these  practices.  SCE  is  a  tool  that  can 
augment  other  acquisition  organization  techniques  for 
discerning  differences  in  the  capabilities  of  offerors. 

The  SEI  has  been  piloting  the  SCE  Method  with  organizations 
in  the  Army,  Air  Force,  and  Navy  since  its  inception  in  1987. 
Several  organizations  have  begtm  to  establish  criteria  for  SCE 
use  which  reflect  the  individual  needs  of  these  organizations, 
and  supplement  the  information  contained  in  this  section  of 
the  guide.  One  major  command  has  drafted  a  policy  requiring 
SCE  use  on  all  MCCR  programs  exceeding  $10  million. 
Another  military  division  is  taking  the  approach  of  requiring 
SCE  use  on  procurements  which  include  software  size 
estimates  greater  than  50  KSLOC.  These  examples  imderscore 
the  importance  of  refining  SCE  usage  criteria  to  best  reflect 
the  acquisition  practices  implemented  at  a  particular 
acquisition  organizational  site.  Different  organizations  have 
different  business  bases,  contractor  communities,  product 
types,  application  domains,  etc.,  all  of  which  affect  site- 
specific  implementing  instructions  for  SCE. 

Pilot  use  of  the  SCE  Method  in  Army,  Navy,  and  Air  Force 
indicates  that  SCE  helps  the  acquiring  organization  in  many 
ways: 

•  Added  software  development  capabjti  _  realism  in  the 
source  selection  process. 

•  Increased  objectivity  in  information  collected  for  an 
acquisition. 

•  Motivation  for  contractor  software  process  improvement 
actions. 
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Software  Development  Capability  Realism:  One  benefit  SCE 
provides  is  the  software  development  capability  realism 
introduced  into  the  proposal  review  and  contractor  analysis 
process.  The  information  SCE  collects  is  timely,  real  and  based 
on  current  projects  and  the  practices  actually  being 
implemented  by  offerors'  engineering  and  managerial 
personnel. 

For  moderate  to  large  software  development  efforts,  a 
currently  popular  means  of  evaluating  a  contractor's  software 
development  abilities  during  a  source  selection  is  the  review 
of  the  offeror's  Software  Development  Plan  (SDP). 
Comparing  the  SCE  findings  with  the  evaluation  of  the 
contractor's  proposal  and  SDP  will  clarify  for  the  program 
office  whether  the  proposed  approach  is  realistic  in  light  of 
the  offeror's  current  process  capability.  Based  on  this 
comparison,  the  program  office  can  better  evaluate  the  risks 
posed  by  each  offeror  and  work  with  the  winning  contractor 
on  a  realistic  software  process  improvement  program. 

Objectivity:  A  second  major  benefit  of  SCE  is  the  objectivity  it 
introduces  in  the  proposal  review  process.  The  SCE  Method 
helps  ensure  an  objective  review  by  putting  a  trained 
evaluation  team  on  site  to  evaluate  the  offerors  activities  and 
compare  them  against  a  public  standard,  the  CMM.  In  the 
typical  source  selection,  evaluating  software  risk  is  a  difficult 
task  because  there  are  few  avenues  for  addressing  this  issue 
other  than  by  evaluating  what  is  in  the  proposal. 

With  the  goals  of  the  CMM  KPAs  as  a  basis,  contractor 
software  process  capability  can  be  reliably  measured  against 
a  common  standard.  This  permits  consistent,  repeatable,  and 
fair  evaluation  of  contractor  software  process  capability.  This 
adds  value  to  the  source  selection  process  by  making  software 
reviews  more  objective. 

Motivation  for  contractor  software  process  improvement 
actions:  In  order  to  remain  competitive  on  successive 
acquisitions,  contractors  must  improve  their  software 
development  processes.  In  contract  monitoring,  capability 
evaluation  can  be  used  to  measure  progress  relative  to  that 
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measured  during  source  selection,  augmenting  an  action  plan 
for  improvement.  The  Performance  Risk  Analysis  Group 
(PRAG)  can  evaluate  process  improvement  based  on  past 
performance  risk  assessments  of  the  software  process. 

By  making  SCE  a  discriminator  in  conducting  acquisitions, 
program  offices  will  motivate  contractors  to  focus  on  software 
process  capability  as  a  means  of  retaining  cr  increasing 
acquisition  agency  specific  business.  Given  the  premise  that 
product  quality  will  follow  process  quality,  focusing  on 
software  process  improvements  resulting  in  increased 
process  maturity  will  increase  the  likelihood  of 

•  Accurate  estimates. 

•  Decreased  variance  among  projects. 

•  Reduced  cost  and  schedule  targets. 

Although  there  is  no  definitive  study  validating  these  benefits 
quantitatively,  there  is  significant  anecdotal  evidence  from 
individual  government  and  industry  organizations  to  suggest 
these  benefits  are  real. 

A  focus  on  improving  software  process  capability  should  lead 
to  error  prevention,  earlier  detection  of  errors  in  the  lifecycle, 
and  an  ability  to  manage  requirements  changes  effectively. 
Improvement  in  processes  which  yield  earlier  detection  of 
errors  and  better  management  of  the  requirements  change 
process  should  save  the  acquisition  agency  money  over  the 
lifecycle  of  major  systems. 

Cost  of  Using  Using  SCE  requires  personnel  and  financial  resources,  on 

SCE  both  the  contractor  and  acquisition  agency  sides.  The  resource 

considerations  affecting  the  implementation  of  SCE  are 

•  Personnel 

•  Time 

•  Financial 

•  Development  organization's  resource  requirements 
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Figure  3-2  shows  the  estimated  acquisition  agency  labor,  in 
person  days,  required  to 

•  Implement  SCE  in  program  documentation. 

•  Train  SCE  team  members. 

•  Conduct  the  site  visits. 

•  Incorporate  the  SCE  findings  into  source  selection  results  / 
decisions. 

The  estimate  assumes  a  single  source  selection,  a  program 
office  having  no  prior  experience  with  SCE,  and  three  offerors 
within  the  competitive  range  who  mxost  be  evaluated.  For  a 
team  of  five  who  conduct  three  site  visits,  the  total  labor  is 
approximately  115  person  days.  For  reference,  the  estimated 
labor  for  an  acquisition  involving  only  one  site  visit  is  65 
person  days  (Total  Effort  Fixed  Costs  39.75  person-days  plus 
Variable  Cost  Effort  of  25  person-days  for  site  visit).  Certainly, 
there  are  economies  of  scale  and  there  are  many  non¬ 
recurring  costs,  such  as  team  training,  which  will  continue  to 
reduce  overall  acquisition  agency  labor  costs  as  trained 
resources  are  utilized  on  subsequent  acquisitions.  In  an 
instance  where  the  program  manager  and  SCE  team  have 
been  trained  and  have  used  the  method  previously,  the 
estimated  labor  to  implement  SCE  on  an  acquisition  (with 
three  site  visits)  declines  to  83.5  person  days.  (114.25,  less  SCE 
information  gathering  5.25,  less  RFP  preparation  1,  less  SCE 
Training  25) 

This  analysis  leaves  it  up  to  members  of  the  individual 
program  office  to  determine  their  own  average  person  cost 
per  day,  average  travel  and  per  diem  costs,  and  subsequently 
add  these  to  the  cost  of  team  training  to  estimate  a  total  dollar 
cost  for  implementing  SCE. 
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Figure  3-2  Estimated  SCE  Labor  For  One  Source  Selection 

The  largest  constraint  on  the  acquisition  agency  is  the  labor 
effort  expended  by  the  individuals  constituting  the  SCE  team. 
This  team  is  needed  for  one  full  work  week  for  every  SCE  site 
visit  that  is  performed.  In  addition,  several  person-days  are 
needed  to  prepare  for  each  site  visit  and  prepare  the  detailed 
report  or  set  of  findings  that  is  submitted  to  the  management 
body  sponsoring  the  evaluation. 

In  addition  to  the  site  visit  requirements,  the  SCE  team  leader 
or  the  program  office's  software  focal  point  will  be  needed  on 
a  part-time  basis  prior  to  the  site  visits  to  incorporate 
appropriate  language  into  the  source  selection  materials  that 
are  affected  by  SCE,  assemble  an  SCE  team,  and  schedule 
training  for  any  untrained  team  members.  This  part-time  task 
will  be  minimal  once  the  respective  acquisition  organization 
has  put  in  place  support  materials  for  SCE,  including  this 
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guide.  After  the  site  visits,  the  SCE  team  leader  will  likely  be 
needed  to  advise  the  evaluation  sponsor  and  perform 
outbriefs  to  the  development  organizations  as  directed  by  his 
or  her  PCO.  Figure  3-3  shows  approximately  the  distribution 
of  human  resources  over  time. 


Team  Preparation 

(one  week) 


Time  (months) 


Figure  3-3  SCE  Person-Loading  Over  Time 

Time  Constraints  The  SCE  team  is  needed  for  at  least  one  and  a  half  weeks  for 

every  a  site  visit.  This  includes 

•  Preparation:  1-2  days. 

•  Travel  time:  1  day. 

•  Site  visit:  3  days  (includes  caucus  and  findings 
preparation). 

•  Time  off  between  site  visits:  1  day. 
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Time  off  is  important  because  site  visits  are  very  intense 
activities.  These  resource  loading  estimates  are  included  in 
Figure  3-3,  above. 

Another  time  constraint  is  imposed  by  the  typical  source 
selection  schedule.  Site  visits  cannot  begin  imtil  after  the 
initial  proposal  evaluation  and  only  on  those  offerors 
remaining  in  the  competitive  range.  This  typically  allows  a 
one  to  two  month  window  for  the  conduct  of  the  on  site  phase 
of  the  SCE.  A  program  manager  does  not  know  the  number  of 
offerors  imtil  proposals  are  received.  This  means  that  the 
program  manager  will  have  to  estimate  how  much  time  is 
needed  to  complete  all  the  SCEs  based  on  the  estimated 
number  of  offerors. 

Figure  3-4  presents  an  actual  cost  summary,  $61,400  (not 
including  labor  cost)  from  a  single  acquisition  that  conducted 
nine  SCE  site  visits.  Another  agency  estimated  the  single 
acquisition  cost  of  using  SCE  to  be  as  low  as  $20,000.  The  SCE 
training  cost  can  be  amortized  over  a  larger  number  of  SCEs 
as  the  individual  team  members  participate  on  other  source 
selections  or  acquisitions.  The  agency  as  a  whole  will  benefit 
over  time  by  reducing  training  costs  as  the  numbers  of  trained 
personnel  increase  and  SCE  use  becomes  more  routine. 

Given  a  $10  million  acquisition,  which  was  introduced  earlier 
as  a  reasonable  threshold  for  seriously  considering  the  use  of 
SCE,  and  a  similar  number  of  offerors  as  shown  in  Figure  3-4, 
the  SCE  cost  is  0.6%  of  the  total  program  cost.  While  using 
SCE  to  help  select  the  most  qualified  offeror  will  not  eliminate 
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cost  or  schedule  problems,  it  will  point  out  the  offeror(s)  with 
the  best  proven  practices  to  mitigate  such  problems,  which 
can  more  than  pay  for  itself  over  the  life  of  the  contract. 


Itemized  Expeneee 

Unit  Costs 
(1  week) 

Total  Costs 
(9  site  visits) 

Travel  Expenses  (5  person  team) 

2,500 

22,500 

Hotel 

1,600 

14,400 

Per  Oiem  and  Miscellaneous 

1,500 

13,500 

One-time  Training  Course 

11,000 

Total  SCE  Costs 

$61,400 

Development 

Organization 

Resource 

Requirements 


Figure  3>4  Example  SCE  Cost  Summary  (Training  Plus  On¬ 
site) 

The  costs  of  supporting  an  SCE  are  significant — ^though  not 
always  as  high  as  those  of  the  acquisition  agency. 
Considerable  preparation  time  is  expended  by  a  company; 
the  company  is  typically  trying  to  put  its  best  foot  forward  for 
the  acquisition  agency,  especially  if  the  SCE  is  done  in 
conjimction  with  a  source  selection.  Thus,  all  development 
organizations  will  perform  some  preparatory  tasks  to 
accommodate  an  SCE. 

Figure  3-5  provides  an  estimate  of  those  costs,  using  $200,000 
as  the  cost  per  person-year  or  $3,850  per  person-week.  The 
preparation  time  of  four  person-weeks  accounts  for  one 
person  full-time  for  four  weeks  or  two  individuals  working 
full-time  for  two  weeks  prior  to  the  SCE  site  visit.  Activities 
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include  identifying  projects  for  review,  getting  maturity 
questionnaires  and  project  profiles  completed,  and 
coordinating  the  site  visit  activities. 


Itemized  Expeneee 

Cost 

Preparation  Time  (4  person-weeks) 

15,400 

Site  Visit  Impact  (1  person-week) 

3,850 

POC  and  Debriefing  Time  (3  person-weeks) 

11,550 

Total  Cost 

$30,600 

Figure  3-5  Example  SCE-Imposed  Development 
Organization  Costs 

The  site  visit  costs  are  those  associated  with  conducting 
individual  interviews.  The  final  costs  are  those  produced  by 
the  offeror  point  of  contact  (PCXI),  who  accompanies  the  SCE 
team,  coordinates  activities  with  the  company,  and  schedules 
the  individuals  for  interviews.  This  PCX!  also  prepares 
individuals  before  each  interview  and  debriefs  the 
interviewee  after  each  interview.  These  costs  vary 
considerably  from  organization  to  organization. 

Costs  can  increase  if  some  contractor  staff  must  travel  to 
another  site  to  accommodate  an  SCE.  Sometimes  the  projects 
selected  for  the  evaluation  are  within  a  product  line  and 
division  that  may  be  at  different  locations.  While  the  SCE 
Method  encourages  project  selection  within  the  same 
geographical  location,  this  cannot  always  be  done  because  of 
the  development  organization's  organizational  make  up. 
Development  organization  personnel  traveling  to 
accommodate  an  SCE  will  not  only  be  spending  travel  funds, 
their  SCE-associated  labor  costs  will  be  greater  as  well.  Under 
these  circumstances,  the  SCE  team  should  work  with  the 
development  organization's  POC  in  an  effort  to  minimize 
impact  on  those  affected  projects. 

The  development  organization's  preparatory  costs  are 
significant:  for  a  period  of  at  least  a  week,  the  offeror's 
operations  will  be  disrupted  by  SCE  site  activities,  company 


Chapter  3;  Deciding  to  Use  SCE 


Suggestions  For 

Optimizing 

Resources 


preparation,  and  debriefing  activities.  These  estimated  costs 
will  change  depending  on  the  contractor,  and  also  as 
contractor  familiarity  with  the  SCE  process  grows  and 
preparation  becomes  more  standardized. 

An  excellent  way  for  an  acquisition  organization  to  optimize 
the  resources  required  for  SCE  implementation  is  to  assign 
full-time  SCE  support  to  acquisitions.  This  option  offers  the 
greatest  savings,  in  both  cost  and  persormel.  Full-time 
support  can  take  on  two  levels  of  involvement.  Persormel  can: 

•  Help  with  the  source  selection  documentation  needed  to 
use  SCE,  identify  team  members,  and  coordinate  their 
training. 

•  Augment  the  SCE  teams  for  specific  acquisitions  by 
participating  in  the  on-site  visits. 

Fully  dedicated  persormel,  who  have  already  gone  up  an  SCE 
learning  curve,  should  be  capable  of  implementing  local  SCE 
policies  and  procedures  quickly  and  effectively,  which  should 
reduce  overall  costs. 

The  use  of  full-time  resources  to  augment  a  program's  SCE 
team  will  insure  organizational  consistency  in  the  practice  of 
the  SCE  Method,  and  assist  the  training  of  persormel  through 
a  form  of  on-the-job  technology  transition.  Utilizing  at  least 
one  full-time  resource  will  act  as  a  significant  acquisition 
"force  multiplier"  when  it  comes  to  implementing  SCE  in  an 
organization. 

The  following  approaches  to  cost  reduction  should  be 
avoided  under  all  circumstances  because  they  would  not 
follow  the  SCE  Method. 

•  Site  visit  time  less  than  three  days. 

•  Teams  of  fewer  than  four  people. 

•  Team  members  untrained. 

•  Using  Maturity  Questiormaire  responses  alone  without 
performing  site  visits. 
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•  Accepting  SPA  results  in  lieu  of  conducting  site  visits. 

These  approaches  undermine  the  consistency,  repeatability, 
and  reliability  of  the  SCE  Method. 
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Chapter  4  Implementing  SCE  in  a  Source 

Selection 


Scheduling  SCE 
in  a  Source 
Selection 


SCE  Schedule 
Leading  Up  To  RFP 
Release 


The  previous  chapter  explored  issues  which  are  relevant  to 
making  the  decision  to  use  SCE  in  an  acquisition.  This  chapter 
will  discuss  the  issues  involved  in  actually  using  SCE  in  the 
source  selection  process.  The  first  section  discusses  the  source 
selection  schedule  and  the  implications  of  using  SCE.  The 
next  section  addresses  the  options  for  factoring  the  SCE 
findings  in  the  source  selection  decision. 

This  section  presents  the  source  selection  process  using  the 
RFP  release  point  as  the  date  from  which  to  build  a  source 
selection  schedule.  The  following  subsections  will  examine 
the  SCE  schedule  within  a  source  selection  before  and  after 
RFP  release.  Each  will  present  a  schedule  of  SCE-related 
activities  showing  a  range  of  time  in  which  the  activities  need 
to  be  completed,  not  the  time  to  complete  the  activity.  These 
schedules  are  approximate  rather  than  absolute,  and  should 
be  tailored  to  the  acquisition's  circumstances.  Each  activity  on 
the  schedules  is  annotated  with  a  comment  describing  the 
activity  and  a  number  which  will  be  referenced  in  the  text  for 
further  discussion  of  each  SCE-related  activity. 

The  schedule  presented  in  Figure  4-1  refers  to  the  major  SCE- 
related  source  selection  activities  that  are  typically 
accomplished  before  the  release  of  the  RFP.  The  first  three 
activities — annotated  with  a  "1,"  "2,"  and  "3" — show  start 
dates  in  the  range  of  seven  or  eight  months  prior  to  the  release 
of  the  RFP.  Depending  on  the  acquisition,  these  dates  could  be 
too  close  or  too  far  from  the  target  RFP  release  date.  Activities 
2  through  5  are  explored  in  more  detail  in  Chapter  5. 

Activity  1 — SCE  Implementation  Planning.  This  is  the  activity 
discussed  in  Chapter  3 — deciding  to  use  SCE  in  an 
acquisition.  This  activity  could  continue  until  the  day  the 
proposals  are  received,  depending  upon  the  proposed 
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application  of  SCE.  Part  of  the  activity  may  include  inserting 
wording  about  planned  SCE  usage  into  the  Commerce 
Business  Daily  (CBD)  annoimcement.  This  activity  starts 
before  the  formation  of  an  SCE  team. 

Activities  2  and  3 — Documentation.  This  activity  involves 
preparing  the  documentation  needed  for  the  Source  Selection 
Plan  (SSP)  and  the  REP.  These  documents  describe  how  SCE 
will  be  used  for  the  acquisition — the  former  for  the  SSA,  SSAC 
and  SSEB,  the  latter  for  the  offeror  community. 


Figure  4-1  Sample  SCE  Schedule  Leading  Up  To  RFP 
Release 


Activity  4 — Pre-Proposal  Conference.  This  is  usually  a  one-day 
event  to  present  the  acquisition  strategy,  contract  type, 
evaluation  criteria,  and  major  program  milestones  to 
prospective  offerors.  It  is  an  opportunity  for  the  offeror 
community  to  hear  first-hand  about  the  pending  program  and 
for  the  government  to  receive  feedback  on  the  program  and 
their  source  selection  approach.  Typically,  a  portion  of  the 
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SCE  Schedule  After 
RFP  Release 


event  will  be  dedicated  to  briefing  how  SCE  will  be  used,  and 
allowing  offerors  to  seek  further  guidance  or  explanation,  if 
needed. 

Activity  5 — Evaluation  Standard  Preparation.  This  activity  is  one 
of  the  most  important  activities  the  SCE  team  leader  or  other 
individual  responsible  will  be  engaged  in  related  to  SCE.  The 
evaluation  standard  will  dictate  to  the  government  team  how 
the  SCE  site  visit  strengths  and  weaknesses  by  key  process 
area  are  measured  and  then  translated  into  findings  used  in 
the  source  selection  decision.  Standards  are  not  provided  to 
the  offerors. 

Activity  6 — SCE  Team  Training  and  Preparation.  This  activity 
will  vary  in  amount  of  work  according  to  the  experience  of  the 
tetim  and  the  SCE  infrastructure  in  place  at  the  acquisition 
organization  that  supports  the  team.  It  is  recommended  that 
team  training  take  place  within  one  to  two  months  of  the 
actual  site  visits.  If  all  members  of  the  team  have  been  trained, 
but  have  not  worked  together  on  an  SCE,  a  practice  SCE  is 
recommended.  All  team  members  should  have  been  trained 
in  SCE  by  the  SEI,  which  is  the  only  official  source  of  training 
at  this  time. 

Figure  4-2  depicts  a  typical  source  selection  schedule  after 
RFP  release.  As  with  previous  schedules,  this  one  is  given  for 
illustrative  purposes  only. 

Activity  1 — Offerors  Prepare  Proposals.  Within  the  acquisition 
organization,  while  offerors  are  preparing  proposals,  the 
month  after  the  RFP  has  been  released  is  spent  preparing  for 
SCE  site  visits.  During  this  period,  the  SCE  team  should  come 
together  to  prepare  for  the  site  visits,  including  team-building 
activities.  The  offerors  will  have  received  instructions  in  the 
RFP  on  exactly  how  to  prepare  for  the  possibility  of  SCE  site 
visits.  This  will  have  included  specifics  regarding  project 
selection,  responding  to  the  maturity  questionnaire,  and 
establishing  a  point  of  contact  who  will  help  with  the  logistics 
of  the  site  visit. 
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Activity  2 — Initial  Evaluations.  After  receipt  of  the  proposals, 
the  technical,  cost,  and  past  performance  PRAG  or  other 
evaluation  teams  begin  their  activities.  The  SCE  team  is 
typically  part  of  the  technical  team  and  may  also  evaluate 
written  proposals  in  software  area(s).  The  receipt  of  proposals 
is  typically  the  initiation  of  the  formal  preparation  for  on-site 
visits  to  the  offerors;  however,  the  visits  themselves  will  not 
be  conducted  until  after  the  competitive  range  determination 
is  made.  The  particular  circumstances  of  the  acquisition  (e.g. 
number  of  offerors,  SCE  team  availability,  competitive  range 
determination)  will  dictate  the  exact  activities  that  occur  for 
the  SCE  team  during  this  period  of  time. 

Activity  3 — SCE  Site  Visits.  The  SCE  team  will  perform  site 
visits  with  all  the  offerors  remaining  in  the  competitive  range. 
Precisely  when  the  SCEs  are  to  be  conducted  is  largely 
dictated  by  how  SCE  is  being  used  in  contributing  to  the 
source  selection  decision  as  described  in  the  SSP  or  evaluation 
plan.  For  instance,  if  the  SCE  results  will  be  factored  into  an 
item  or  items  of  specific  criteria,  they  should  be  conducted 
after  receipt  of  change  requests  (CRs)  or  deficiency  reports 
(DRs)  but  before  face-to-face  discussions.  If  SCE  is  to  be  used 
to  support  PRAG  (peist  performance)  findings,  then  site  visits 
can  be  accomplished  anytime  after  competitive  range 
determination  but  before  best  and  final  offers  (BAFOs)  are 
issued. 

Activity  4 — Discussions  /  Negotiations.  This  activity  addresses 
that  part  of  the  process  where  CRs,  DRs,  and  Points  For 
Negotiation  (PFNs)  are  communicated  to  the  offerors  within 
the  competitive  range.  These  tools  can  be  used  by  the 
government  to  communicate  SCE  findings  to  the  offerors.  The 
government  allows  all  remaining  offerors  to  respond  to  each 
and  then  requests  these  offerors  to  submit  a  BAFO.  The 
government  will  also  begin  developing  "model"  contracts  for 
those  offerors  still  witirin  the  competitive  range  to  reflect  the 
terms  and  conditions  agreed  upon  by  both  parties  (the 
government  and  that  particular  offeror).  Offerors  are  advised 
that  any  deviation  from  the  agreed-to  terms  and  conditions 
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that  are  not  traceable  from  their  original  proposal  may  result 
in  their  proposal  being  unacceptable.  This  period,  too,  can  last 
more  or  less  time  than  depicted  in  Figure  4-2. 


Figure  4-2  Sample  SCE  Schedule  After  RFP  Release 

Activity  5 — BAFO  Evaluations.  Here,  the  government 
considers  the  offerors'  BAFOs.  This  may  entail  significant 
analysis  comparing  the  offeror's  responses  as  to  their  effect 
upon  the  analysis  and  determinations  formulated  up  to  this 
point.  Here  again  the  new  or  revised  information  is  analyzed/ 
evaluated  against  the  approved  evaluation  standards  used  in 
evaluating  the  offerors  initial  proposal. 

Activity  6 — Source  Selection  Decision.  Once  all  of  the 
aforementioned  activities  have  been  completed,  an  award 
decision  will  be  made.  The  SSA  will  have  been  convinced  that 
an  equitable,  effective  and  objective  evaluation  of  each 
offeror's  strengths,  weaknesses,  and  improvement  activities 
has  been  made  by  the  SSEB/T.  The  time  from  receipt  of 
BAFOs  to  contract  award  can  take  some  time  as  there  are  a 
considerable  number  of  legally  imposed  actions  that  must 
take  place  before  announcing  an  award. 
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Using  SCE 
Results  in  the 
Source 
Selection 


This  section  explores  using  the  findings  from  an  SCE  site  visit 
in  the  final  decision  of  a  source  selection.  Section  M  of  the  RFP 
notifies  offerors  that  the  use  of  SCE  would  be  evaluated  as  a 
specific  criterion.  Included  in  this  section  will  be  an  example 
using  a  color  code  scheme  as  the  rating  tool  in  the  source 
selection  process.  The  discussions  that  follow,  while  using 
data  from  real  acquisitions,  have  been  edited  to  eliminate 
source  selection  sensitivity  or  to  illustrate  a  key  point  about 
SCE  implementation.  A  reference  to  the  PRAG  is  included  at 
the  end  of  this  section.  SCE  findings  would  be  incorporated 
into  the  performance  risk  assessment  report/briefing  if  SCE  is 
used  as  part  of  PRAG  activities. 

There  is  a  significant  difference  between  specific  criteria  and 
performance  risk  assessments.  The  source  selection-related 
regulations,  regardless  of  the  implementing  agency,  require 
that  specific  criteria  encompass  the  characteristics  of  the 
program  being  acquired.  All  acquisition  agencies  require  the 
establishment  of  order  of  precedence  for  the  various  specific 
criteria,  so  that  the  offerors  understand  their  relative 
importance  and  can  craft  their  proposal  accordingly. 

Additionally,  pre-established  standards  of  evaluation  are 
prepared  for  each  criterion  and  the  offerors'  proposal  is 
measured  against  those  standards  by  the  SSEB.  This 
evaluation  against  the  evaluation  standards  then  forms  the 
basis  of  comparison  of  one  proposal  to  another,  w  l  ach  is  done 
in  a  source  selection,  typically  by  a  more  senior  body,  such  as 
the  Source  Selection  Advisory  Council. 

Note  that  in  developing  any  evaluation  standards  (Figure  4-4 
and  Figure  4-5),  the  appropriate  procurement  regulations 
should  be  followed  as  well  as  consulting  and  working  with 
the  source  selection  staffs. 

To  get  the  most  emphasis  of  SCE  use  in  source  selection,  SCE 
should  be  used  as  a  specific  criterion  and  may  also  be 
evaluated  by  the  PRAG  for  performance  risk.  Use  of  SCE 
results  as  specific  criterion  and/or  in  the  PRAG  for 
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Using  SCE  as  a 
Specific 
Criterion  For 
Award 


performance  risk  will  be  decided  by  the  SSA  at  the  same  time 
the  SSP  is  approved,  based  on  source  selection  regulations 
and  program  requirements. 


Each  offeror's  proposal  will  be  evaluated  against  the 
following  areas  listed  in  descending  order  of  importance  (list 
areas  in  descending  order  of  importance  or  specify  relative 
importance.  Note:  Areas  should  be  limited  to  two  [including 
cost/price],  when  feasible). 

The  technical  area  (or  each  of  the  areas  [except  cost /price]  if 
more  than  two  areas  used)  will  be  rated  in  three  ways: 

1.  A  color/adjectival  rating. 

2.  A  proposal  risk  rating. 

3.  A  performance  risk  rating. 

The  color/adjectival  rating  depicts  how  well  the  offeror's 
proposal  meet®  the  evaluation  standards  and  solicitation 
requirements.  Proposal  risk  assesses  the  risk  associated  with 
the  offeror's  proposed  approach  as  it  relates  to  accomplishing 
the  requirements  of  this  solicitation.  Performance  risk 
assesses  the  probability  of  the  offeror  successfully 
accomplishing  the  proposed  effort  based  on  the  offeror's 
demon-strated  present  and  past  performance.  The 
government  will  conduct  a  p)erformance  risk  assessment 
based  on  the  offeror's  relevant  present  and  past  performance. 
In  assessing  this  risk,  the  government  will  use  performance 
data  to  evaluate  the  areas  listed  above. 

Offerors  are  to  note  that  in  conducting  the  performance  risk 
assessment,  the  government  will  use  both  data  provided  by 
the  offeror  and  data  obtained  from  other  sources.  Within  each 
area  (other  than  cost/ price),  each  of  the  three  ratings — color/ 
adjectival,  proposal  risk,  and  performance  risk — will  be 
cortsidered  in  making  an  integrated  source  selection  decision 
as  to  which  proposal  is  most  advantageous  to  the 
government. 
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SCE  should  be  used  as  an  item  under  an  area  of  specific 
criterion  such  as  Technical /Management  and/  or  in  the 
PRAG  for  performance  risk  assessments.  Ultimately,  how 
SCE  findings  are  translated  into  SCE  results  and  used  in  the 
Source  Selection  (SS)  should  be  determined  by  the  SSA  based 
on  source  selection  regulations  and  program  requirements. 
Figure  4-3  provides  an  illustration  from  an  acquisition 
employing  SCE  as  a  technical  item  (software  engineering 
capability)  in  the  technical  area. 


Technical  Area 


Software  Engineering  Capability 
Item 


Technical  Approach  Item 


Management  Item 


Description 

The  acquisition  organization  will  evaluate  the  offeror’s  software 
process  by  reviewing  its  Software  Process  Improvement  Plan  (SPIP) 
and  by  conducting  an  on-site  visit  using  the  Software  Engineering 
Institute-  (SEi)  developed  Software  Capability  Evaluation  (SCE) 
Method. 


The  government  will  evaluate  the  offeror’s  technical  approach  to 
accomplishing  the...  tasks.  The  evaluation  will  assess  the  extent  that 
the  approach  satisfies  the  objective,  goals,  and  requirements  of  the 
Statement  of  Work. 


The  acquisition  organization  will  evaluate  the  offeror’s  managerrwnt 
approach  to  accomplish  contract  goals  and  the  extent  to  which  the 
approach  achieves  the  objectives,  goals,  and  requirements  of  the 
solicitation.  The  government  will  focus  on... 


Figure  4-3  Sample  Set  of  Specific  Criteria  or  Technical 
Items 

What  follows  is  an  example  using  SCE  as  a  specific  criterion 
in  making  the  source  selection  decision.  The  specific  needs  of 
the  acquisition  should  dictate  the  exact  approach  to  be  used. 
In  this  example,  the  items  of  the  technical  area  are  listed  in 
descending  order  of  importance.  This  example  is  but  one 
approach  and  method  for  implementing  SCE  findings  in  the 
source  selection  decision. 

This  example  continues  the  discussion  of  SCE  as  a  specific 
criterion  as  depicted  in  Figure  4-3.  The  example  will  illustrate 
the  incorporation  of  the  SCE  findings  into  the  various  source 
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Color  Coding  the 
Technical  Items 


selection  evaluation  tools /documents  that  are  used  for  the 
source  selection  as  well  as  the  definitions  established  for  the 
various  color  ratings  and  the  identification  of  risk. 

Applying  color  codes  begins  when  the  SCE  team  has 
completed  all  site  visits  and  the  evaluations  of  the  offerors' 
Software  Process  Improvement  Plans  (SPIP).  In  this  example 
the  SPIP  was  requested  to  be  prepared  and  submitted 
separately  at  the  same  time  the  proposal  was  submitted. 

Using  this  approach,  each  technical  item  is  assigned  a  color 
which  corresponds  to  a  rating — from  "exceptional"  to 
"unacceptable."  For  each  item,  an  evaluation  standard  is 
written  to  define  precisely  what  an  offeror  must  do  to  be 
assigned  a  certain  color. 

Figure  4-4  shows  how  colors  were  used  and  how  ratings  such 
as  "exceptional"  were  defined  [USAF  88]. 


Color 

Rating 

Definition 

Blue  (B) 

Exceptional 

Exceeds  specified  performance  of  capability  in  a 
beneficial  way  to  the  government.  Has  high 
probability  of  satisfying  the  requirement.  Has  no 
significant  weakness. 

Green  (G) 

Acceptable 

Meets  evaluation  standards.  Has  good  probability 
of  satisfying  the  requirement.  Any  weakness  can  be 
readily  corrected. 

Yellow  (Y) 

Marginal 

Fails  to  meet  evaluation  standards  or  has  low 
probability  of  meeting  the  requirement;  or  has 
significant  but  correctable  deficiencies. 

Red(R) 

Unacceptable 

Fails  to  meet  a  minimum  requirement.  Deficiency 
requires  a  major  revision  to  correct. 

Figure  4-4  Description  of  Coiors 

Along  with  each  color,  the  evaluation  team  assigns  a  risk 
rating  which  reflects  the  risk  associated  with  the  offeror 
performing  on  time,  within  budget,  and  within  the  specified 
performance  parameters.  Figure  4-5  lists  the  ratings  and  their 
definitions.  This  example  used  the  consistency  between  the 


CMU/SEI-94-TR-5 


4-69 


Chapter  4:  Implementing  SCE  in  a  Source  Selection 


SCE  findings  and  the  achievability  of  the  offeror's  software 
process  improvement  program  to  denote  the  nsk  of  the  item. 
Software  Engineering  Capability. 


Risk 

Definition 

High  (H) 

Likely  to  cause  significant  serious  dismption  of 
schedule.  Increase  in  cost,  or  degradation  of 
performance  even  with  special  contractor  emphasis 
and  close  government  monitoring. 

Moderate  (M) 

Can  potentially  cause  some  disruption  of  schedule, 
increase  in  cost,  or  degradation  of  performance. 
However,  special  contractor  emphasis  and  close 
government  monitoring  will  probably  be  able  to 
overcome  difficulties. 

Low(L) 

Has  little  potential  to  cause  disruption  of  schedule, 
increase  in  cost  or  degradation  of  performance. 
Normal  contractor  emphasis  and  government 
monitoring  will  probably  be  able  to  overcome 
difficutties. 

Figure  4<5  Description  of  Risks 


A  complete  set  of  offeror  findings  (strengths  and  weaknesses) 
measured  against  the  CMM  KPAs,  should  be  used  in 
assigning  color  codes  and  risks.  The  SCE  team  should  provide 
the  SSEB  with  these  findings.  See  Figure  4-6,  Figure  4-7,  and 
Figure  4-8  for  an  example.  (Figure  4-6  is  a  summary  of  the 
findings,  while  Figure  4-7  and  Figure  4-8  show  the  details  of 
that  summary.) 
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Summary  Reaulta 

Strong 

•  Requirements  Management 

•  Peer  Reviews 

•  Software  Project  Tracking  and  Oversight 

Acceptable 

•  Software  Project  Planning 

•  Software  Configuration  Management 

•  Software  Quality  Assurance 

•  Training  Program 


Weak 

•  Organization  Process  Focus 


Figure  4^  Summary  Findings  From  a  Recent  SCE 


The  source  selection  organization  should  at  no  time  ask  for  or 
accept  findings  from  a  Software  Process  Assessment  (SPA). 
As  discussed  in  Chapter  1,  SPA  findings  are  determined  for  a 
different  purpose  and  are  inappropriate  for  use  with  SCE 
findings  in  a  source  selection  decision. 

The  summary  findings  shown  in  Figure  4-6  reveal  that  only 
one  key  process  area  was  weak.  The  weaknesses  contributing 
to  that  determination  can  be  found  in  Figure  4-7  and  Figure  4- 
8.  Although  there  were  weaknesses  in  other  key  process  areas, 
only  the  Organization  Process  Focus  weaknesses  were  found 
to  be  significant  enough  for  that  KPA  to  be  included  in  the 
summary  findings  weak  area.  The  details  of  that 
determination  are  made  by  the  SCE  team  in  the  context  of  this 
specific  acquisition.  This  means  that  the  SCE  team  used  their 
individual  professional  judgment  to  determine  the  degree  of 
satisfaction  of  the  goals  of  each  KPA.  The  context  of  these 
determinations  is  critical  to  the  findings.  For  example,  it  is 
possible  that  an  organization  may  have  a  software 
configuration  management  system  that  most  experienced 
personnel  would  consider  excellent.  However,  the  SCE  team 
may  have  foimd  that  one  project  does  not  use  it,  another 
project  uses  it  very  effectively,  and  a  third  or  fourth  project 
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may  use  it  in  differing  levels  of  application.  This  is  an  example 
where  the  SCE  team  would  be  faced  with  determining,  from 
the  organizational  standpoint,  whether  a  finding  for  the 
Software  Configuration  Management  KPA  is  acceptable, 
weak  or  strong.  On  one  hand  it  was  determined  that  the 
configuration  management  system  in  place  is  excellent  (a 
strength),  on  the  other  hand  the  evidence  suggests  spotty 
implementation  and  or  application  (acceptable  or  weak?). 
Does  this  mean  the  finding  is  reported  as  a  strength, 
acceptable  or  as  a  weakness?  This  type  of  dilemma  is  typical 
of  those  faced  by  the  SCE  team  for  which  the  various 
background  experience  in  the  different  disciplines  comes  into 
play  in  providing  consensus  from  a  professional  judgment 
standpoint  on  specific  findings  for  each  KPA  investigated. 
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Figure  4>7  Detailed  Findings 
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& 


Software  Project  Planning 

Strengths 

•  Procedure  for  sizing  and  costing  of  software 
exists  project  to  project 

•  Extensive  collection  of  management  metrics 
and  tracking  of  progress 

•  Schedule  and  performance  based  on  real 
progress 

•  Cost  and  schedule  consistent  with  size 

Weaknesses 

•  Inconsistent  sizing  procedure  across  organiza¬ 
tion 

•  Lack  of  completely  written  sizing  procedure 

Improvement  Activities 

none  noted 


S/W  Configuration  Management 

Strengths 

•  Effective  change  control  process 

•  Automated  tool  to  enforce  change  control 
process 

•  Effective  traceability  between  development 
products 

Weaknesses 

•  Lack  of  mechanism  insuring  the  adequacy  of 
regression  testing 

improvement  Activities 

none  noted 


IS 


Training  Program 

^  Strengths 

•  Training  database  by  individual  exists 

•  Many  diverse  courses  offered 

^  •  Individualized  training  program  updated  during 
yearly  appraisal 

Weaknesses 

•  Training  program  inconsistently  implemented 
and  emphasized  across  the  organization 

•  Inadequate  resources  to  ensure  timely  training 

..  Improvement  Activities 

none  noted 


Organization  Process  Focus 
Strengths 

•  Organizational  function  exists 

•  Full-time  resources  in  place 

•  Organizational  focus  for  metrics  collection 

Weaknesses 

•  Lack  of  buy-in  from  the  engineering  staff  (many 
unaware  of  existence) 

•  Lack  of  SEPG  focus  and  record  of  accomplish¬ 
ment 

Improvement  Activities 
none  noted 


Figure  4-8  Detailed  Findings  (continued) 


Another  aspect  of  using  SCE  is  illustrated  by  the  use  of  PFNs 
to  communicate  software  process  weaknesses  identified  by 
SCE  to  the  offerors  within  the  competitive  range.  A  CR  should 
be  used  to  communicate  a  weakness  initially.  A  PEN  can  be 
used  to  identify  those  points  the  government  wishes  to 
discuss  further.  A  PEN  or  CR  will  never  be  used  to  identify  a 
deficiency.  The  SSEB  then  considers  their  responses  with  the 
original  SCE  findings  before  making  a  final  determination 
against  the  evaluation  standard.  This  approach  allows  the 
offerors  the  opportunity  to  point  out  any  oversights  on  the 
part  of  the  SCE  team.  The  SCE  team  could  prepare  a  PEN  (or 


4-74 


CMU/SEI-94-TR-5 


Chapter  4:  Implementing  SCE  in  a  Source  Selection 


CR  if  appropriate)  to  let  offerors  know  what  weaknesses  were 
found.  Figure  4-9  is  an  example  of  a  PFN.  This  example  details 
the  specific  weaknesses  found  by  the  SCE  team  that  made  the 
KPA  Organization  Process  Focus  weak. 


Source  Selection  Infonnation  (See  FAR  3.104) 
For  Official  Use  Only  (when  filled  in) 


POINT  FOR  NEGOTIATION 


1 


Government  Reference: 
IFPP  Paragraph  3.3.4 


Offeror 

XYZ  Corporation 


"3 


Offeror  Reference: 


Register  Number 
PFN-XYZ-S-001 


Point  for  Negotiation: 

The  key  process  area  (KPA)  found  by  the  Software  Capability  Evaluation 
(SCE)  to  be  weak  is  Organization  Process  Focus.  The  detailed  findings  leading 
to  this  conclusion  are  as  follows: 

•  Lack  of  buy  in  from  the  engineering  staff  (many  unaware  of  existence) 

•  Lack  of  SEPG  focus  and  record  of  accomplishment 


Mr 


SCE  team  Chief: 


SSEB/T  Chairperson: 


Source  Selection  Information  (See  FAR  3.104) 
For  Official  Use  Only  (when  filled  in) 


Figure  4-9  Findings  Incorporated  Into  a  Point  For  Negotia¬ 
tion  (PFN) 


The  findings  that  go  into  a  PFN  may  vary.  One  acquisition 
organization's  approach  was  to  provide  only  those 
weaknesses  in  the  PFN  that  caused  an  entire  key  process  area 
to  be  evaluated  as  weak  (as  in  Figure  4-6).  Those  are 
significant  weaknesses  which,  depending  upon  the  affected 
key  process  area,  may  influence  the  evaluation  standard  one 
way  or  another.  Alternatively,  the  entire  findings  set  may  be 
communicated  in  the  PFN.  In  deciding  what  to  include  in  the 
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PFN/CR,  the  SCE  team  leader  should  work  very  closely  with 
the  PCO,  SSEB  chairperson,  and  the  acquisition 
organization's  legal  advisor. 

A  PFN/CR  is  a  way  to  communicate  an  SCE  weakness(es)  to 
an  offeror  and  allow  the  offeror  to  respond  with  one  of  the 
following; 

•  Evidence  showing  the  government's  SCE  team  made  an 
oversight. 

•  A  response  accepting  the  findings. 

•  A  response  accepting  the  findings  and  identifying 
improvement  activities  to  remedy  the  weaknesses. 

•  A  combination  of  the  above  previous  responses. 

A  cover  letter  sent  with  the  PFNs/CRs  will  explain  how  the 
offeror  may  respond.  It  is  recommended  that  the  letter 
include  a  page  limitation  for  the  offeror's  response  so  that  the 
SSEB  is  provided  with  only  relevant  evidence. 

When  the  responses  to  the  PFNs/CRs  have  been  received 
from  the  offerors  (t5rpically  five  to  seven  days  are  allowed  for 
responses)  the  SCE  team  leader  should  analyze  them  to  see  if 
material  changes  in  the  findings  are  required  that  would 
necessitate  recalling  the  SCE  team.  The  only  time  the  SCE 
team  would  reverse  a  decision  on  a  finding,  is  if  the  offeror 
shows  proof  that  the  team  overlooked  something. 

The  SCE  team  performs  an  analysis  and  makes  any  final 
adjustments  to  the  findings.  These  findings  will  be  factored 
into  the  technical  area /item  evaluation  results  for  each 
offeror.  The  manner  in  which  SCE  findings /results  are 
factored  into  the  source  selection  results  depends  upon  how 
SCE  was  structured  into  the  source  selection  (e.g.  items, 
factors  etc).  Your  PCO  and  procurement  regulations  will 
guide  you  through  this  step.  Figure  4-10  and  Figure  4-11 
provide  an  example  item  summary  for  the  set  of  findings 
shown  in  Figure  4-6,  Figure  4-7,  and  Figure  4-8.  The  example 
assumes  that  no  changes  to  the  findings  were  made  during 
the  PFN/CR  process. 
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The  item  summary  contains  the  color  rating  and  associated 
risk  for  the  respective  offer,  some  background  on  the  projects 
the  SCE  evaluated,  the  summary  and  detailed  findings  made 
by  the  SCE  team  while  on  site,  and  a  statement  justifying  the 
assigned  risk.  In  order  to  determine  the  color  rating,  the  SCE 
team  applied  the  findings  to  the  evaluation  standard. 
Similarly,  for  this  example,  the  risk  was  assigned  based  upon 
consistency  between  the  offeror's  communicated  capability 
found  in  the  SPIP  and  the  actual  SCE  findings.  In  this 
example,  the  offeror  was  rated  blue  with  a  low  risk.  The  item 
summary  then  points  out  the  various  strengths  and 
weaknesses  in  their  appropriate  location  and  justifies  the  risk 
rating. 

At  this  level  of  evaluation,  within  the  SSEB,  the  offerors  are 
only  compared  to  a  pre-established  standard.  No  offerors  are 
compared  to  one  another. 
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Source  Selection  Information  (See  FAR  3.104)  —  For  Official  Use  Only  (When  Filled  in) 


Hern  Summary 

Area;  Technical 

Item;  T3/Software 
Engineering  Capability 

Offeror;  XY2  Coip 

Color  Rating;  Blue 

Description  of  Proposal 

The  offeror  proposed  a  software  PIP  which... 


The  software  Process  Improvement  Plan  was  found  to  be  consistent  with  the  SCE  findings.  The  offeror’s 
program  of  software  process  improvement  is  genuine,  with  considerable  emphasis  on  organizational 
standardization  and  removal  of  defects  through  rigorous  reviews.  The  projects  examined  as  part  of  the 
Software  Capability  Evaluation  (SCE)  are  as  follows; 


ABCD 


•  HAVE  GOLD  PLATE  •  COBRA  LIBRARY  •  CCXYZ 


Strengths 

Requirements  Management 

•  Defined  review/status  mechanism  in  place 

•  Very  clear,  strong  lines  of  authority 

•  Software  engineering  represented  throughout  system  engineering  process  (and  vice  versa) 

•  Action  items  tracked  to  closure  by  management 

•  Sure  technical  presence  at  management  level 

Software  Project  Tracking  and  Oversi^t 

•  Provides  wide  coverage  of  software  process  at  a  detailed  level 

•  Extensive  use  of  programmers  notebooks  to  guide  staff  through  phases  of  the  process 

•  Firm  emphasis  on  populating  useful  software  development  folders 

Peer  Reviews 

•  Multiple,  rigorous  requirements,  design,  and  code  inspections  conducted 

•  Training  required  to  participate  on  peer  reviews 

•  Experienced,  senior  people  lead  reviews 

•  Currently  tracking  defects  and  beginning  to  analyze  results 


Item  Chief  Signature; 


Area  Chief  Signature; 


Figure  4-10  Findings  Incorporated  in  Item  Summary 
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I  _ 

Acceptable  Points 

SQA 

•  Experienced  personnel  and  independent  reporting  chain 
Software  Project  Planning 

•  Procedure  for  sizing  and  costing  software  exist  project  to  project 

•  Extensive  collection  of  management  metrics  and  tracking  of  progress 

SCM 

•  Effective  change  control  process  and  tracesdsility  between  development  projects 
Training  Program 

•  Solid  emphasis  from  management  and  extensive  in-house  software  courses 

•  SEPG 

•  An  organizational  function  exists  with  full-time  resources  in  place 

Weaknesses 

The  Key  Process  Area  found  by  the  Software  Capability  Evaluation  to  be  weak  was: 
Organization  Process  Focus 

•  Lack  of  buy-in  from  the  engineering  staff  (many  unaware  of  existence) 

•  Lack  of  SEPG  focus  and  record  of  accomplishment 


Overall  Risk  Assessment  and  Evaluation  Summary 

Low  Risk:  The  consistency  between  their  SCE  findings  and  software  process  improvement  plan  shows 
they  understand  their  current  maturity  level  and  where  they  are  going  as  an  organization.  They  are  very 
strong  technically  (very  close  to  being  strong  in  alt  the  key  process  areas)  and  are  committed  to 
developing  quality  software  using  a  continually  improving  development  process.  This  contractor's 
commitment  to  process  improvement  was  further  evidenced  by  the  process  rigor  in  place  on  one  of  their 
commercial  programs  where  no  development  standards  were  required.  Their  process  was  still  the  same 
and  management  exercised  the  same  controls. 


Figure  4-1 1  Findings  Incorporated  in  Item  Summary 
(continued) 

Item  Summaries  are  reviewed  by  the  SSEB/T  chairperson  and 
then  an  area  summary  is  prepared  which  normally  "rolls  up" 
all  (or  most)  of  the  strengths  and  weaknesses  from  the 
individual  item  summaries  and  then  identifies  an  overall  risk 
for  that  area.  This  information  is  reviewed  by  the  PCO,  legal 
representatives,  and  the  SSAC.  The  legal  and  PCO  review  will 
examine  everything  to  insure  that  the  evaluation  standards 
have  been  consistently  applied  and  that  the  item  and  area 
summaries  contain  consistent  types  and  levels  of  information. 
The  SSEB/T  will  present  this  information  to  the  SSAC.  The 
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SSAC  members  will  analyze  the  SSEB/T's  evaluation  results 
and  start  the  process  of  comparing  each  offerors  strengths, 
weaknesses  and  risk — an  activity  the  SSEB  is  not  allowed  to 
do. 

In  parallel,  the  SSEB  will  make  a  formal  presentation  to  the 
SSAC  outlining  the  color  codes,  strengths,  weaknesses  and 
risks  for  each  offeror  for  each  item  and  area  resulting  from 
their  evaluations.  During  this  presentation,  the  SCE  team 
leader,  as  a  member  of  the  SSEB,  should  be  prepared  to 
elaborate  on  any  of  the  findings  from  any  of  the  offerors.  For 
example,  the  SCE  team  leader  should  be  prepared  to  explain 
not  only  why  an  offeror  was  weak  in  software  configuration 
management,  but  also  why  the  SCE  team  found  their  change 
control  process  lacking.  The  SSAC  will  want  to  ensure  that  the 
SSEB  can  substantiate  their  findings  with  documented 
evidence. 

At  this  point  in  the  source  selection  process,  the  SSAC,  after 
completing  their  comparative  analysis  of  all  final  proposals' 
strengths,  weaknesses  and  risks,  may  elect  to  assign  a 
different  color  code  separate  from  the  SSEB  or  it  may  ask  the 
SSEB  to  reconsider  its  color  codes  in  light  of  information 
discussed  in  the  SSAC  briefing.  These  actions  are  normally 
done  on  an  "exception"  basis  and  are  not  common  since  the 
SSAC  would  have  reviewed  this  material  at  the  time  of 
competitive  range  and  before  BAFOs  were  issued;  therefore, 
any  "disconnects"  should  be  resolved  before  BAFOs  are 
received.  Unless  an  offeror  completely  changes  its  proposal 
approach,  there  should  be  no  "surprises"  in  the  BAFOs.  Ihe 
SSAC  will  ensure  that  the  evaluation  for  each  criterion  has 
been  consistently  and  fairly  applied  to  all  offerors. 

Figure  4-12  shows  one  way  the  findings  from  a  series  of  SCEs 
has  been  presented  formally  to  a  SSAC.  Each  offeror's 
technical  rating,  strengths  and  weaknesses,  risk,  and  a 
summary  explaining  the  basis  for  the  risk  are  identified  and 
placed  next  to  the  other  offerors  so  that  the  SSAC  may 
compare  and  discuss  them  during  a  presentation.  This 
normally  represents  the  lowest  level  of  detail  presented  to  the 
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SSAC  during  the  formal  presentation.  It  is  during  this 
presentation  that  an  SCE  team  leader  may  have  to  articulate 
why  certain  key  process  areas  were  a  strength  or  weakness  for 
a  particular  offeror.  The  expertise  of  the  SCE  team  leader  is 
needed  to  commxmicate  why  a  KPA  was  strong  or  weak  and 
its  significance  within  the  software  process. 


Item:  T-3  Software  Engineering  Capability 

Offeror  A 

Offeror  B 

Offeror  C 

Blue 

Yellow 

Yellow 

Strength 

•  Requirements 

Management 

•  Peer  Reviews 

•  Software  Project  Tracking 
and  oversight 

•  None 

•  Organization  Process 

Focus 

Weakness 

•  Software  Engineering 
Process  Group 

•  Organization  Process 

Focus 

•  Software  Quality 

Assurance 

•  Training  Program 

•  Peer  Reviews 

•  Software  Project  Tracking 
and  Oversight 

•  Training  Program 

Risk 

Offeror  is  very  strong 
technically  and  is  committed 
to  developing  quality 
software  using  a 
continuously  improving 
development  process 

Because  of  the  iarge 
disparity  between  SCE 
findings  and  their  submitted 
SPIP,  it  is  highly  questionable 
whether  the  software  process 
improvement  is  being 
implemented 

Offer’s  SPIP  indicated  they 
are  at  the  initial  maturity  level 
with  their  best  practices 
being  applied  to  all  new 
programs 

L 

H 

M 

_ 

Figure  4-12  Findings  Output  From  the  Evaiuation  Standard 

The  SCE  written  report  must  also  back  up  and  provide 
substantiation  or  articulate  reasons  for  the  ratings'  assigned 
since  the  briefing  is  reduced  to  "bullets"  only  and  should  be 
derived  from  the  detailed  written  findings. 

Figure  4-12  illustrates  how  risk  was  assigned  to  the  software 
engineering  capability  technical  score  (color  rating)  in  a  recent 
source  selection.  Note  that  Offerors  B  and  C  have  yellow  as 
their  technical  score,  but  Offeror  B  has  a  high  risk  and  Offeror 
C  has  a  moderate  risk;  yet  Offeror  C  has  three  weak  Key 
Process  Areas  and  Offeror  B  has  only  two.  How  did  this 
occur? 
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Risk  in  this  acquisition  was  assigned  based  upon  the 
consistency  of  the  organization's  process  improvement 
program  and  the  SCE  findings,  because  it  was  stated  clearly 
in  the  RFP  for  this  acquisition  that  an  organization  could  be  at 
the  Initial  maturity  level  and  still  be  awarded  the  contract.  It 
was  also  stated  in  the  Instructions  for  Proposal  Preparation 
(IFPP)  that  risk  would  be  used  as  a  measure  of  an 
organization's  process  improvement  realism.  If  an 
organization  had  a  realistic  program  of  software  process 
improvement,  then  they  were  considered  low  risk,  regardless 
of  their  current  maturity  level  rating.  If  an  offeror  claimed  to 
be  at  the  Defined  or  Managed  maturity  level  in  its  SPIP,  but 
the  SCE  findings  showed  the  offeror  to  be  at  the  Initial  or 
Repeatable  maturity  level,  then  the  SSEB  would  assign  either 
a  high  or  moderate  risk.  This  assignment  depended  upon  the 
magnitude  of  the  disparity  between  the  SPIP  and  the  SCE 
findings. 

Offeror  B  had  identified  itself  as  being  at  the  Defined  maturity 
level  and  did  not  have  an  improvement  plan  that  would 
substantiate  its  progress  through  the  lower  maturity  level. 
The  SSEB /SCE  team  determined  Offeror  B  to  be  closer  to  the 
Initial  maturity  level.  In  short.  Offeror  B  was  imaware  of  its 
actual  lower  maturity  level  and  was  consequently  assigned  a 
high  risk  with  only  two  weak  key  process  areas,  while  Offeror 
C  received  a  moderate  risk  with  three.  Offeror  C,  on  the  other 
hand,  had  a  realistic  SPIP  indicating  it  was  at  the  Initial 
maturity  level  with  its  best  practices  being  applied  to  all  new 
programs.  The  SCE  findings  confirmed  this  and  resulted  in 
assigning  a  lower  risk  to  this  offeror. 
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Area:  Technical 

Offeror 

A 

Offeror 

B 

Offeror 

C 

Software  Engineering 
Capability 

Color 

Blue 

Yellow 

Yellow 

Risk 

L 

H 

M 

Technical  App. 

Color 

Green 

Yellow 

Blue 

Risk 

L 

L 

L 

Management 

Color 

Green 

Green 

Blue 

Risk 

L 

L 

L 

SUMMARY  RESULTS 

Color 

Green 

Yellow 

Green 

Risk 

L 

H 

M 

Figure  4-13  Technical  Area  Summary 


The  last  step  of  the  process  is  the  integration  of  the  SCE 
technical  rating  and  risk  factor  with  those  of  the  other 
technical  items  to  produce  a  technical  area  summary,  as 
shown  in  Figure  4-13.  At  this  point,  the  SSAC  will  integrate 
the  color  codes  and  risk  factors  into  area  summaries  based 
upon  their  own  analysis  and  presents  them  to  the  SSA.  The 
SSAC  then  conducts  a  comparative  analysis  of  all  offerors' 
strengths,  weaknesses  and  risks  as  presented  by  the  SSEB/T 
on  these  item  and  area  simunaries  and  presents  it  to  the  SSA. 
Note:  SSAC  does  not  make  written  recommendations  to  the 
SSA.  Note  that  in  this  example  the  items  in  the  Technical  Area, 
Management,  Technical  Approach  and  Software  Engineering 
Capability  are  listed  in  descending  order  of  importance.  This 
illustration  of  risk  identification  and  assessment  is  not  the  sole 
method  for  approaching  the  risk  problem.  Acquisitions 
should  tailor  the  risk  assignment  to  the  specific  needs  of  the 
acquisition. 
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Performance 
Risk  Analysis 
Group  (PRAG) 


Offerors  past  aitd  present  performance  is  evaluated  by  the 
PRAG.  Their  results  will  be  presented  to  the  SSA  in  the  form 
of  performance  risk. 

Performance  Risk  Assessment  Definitions: 

High 

Significant  doubt  exists,  based  on  offeror's  performance 
records,  that  the  offeror  can  perform  the  proposed  effort. 


Moderate 

Some  doubt  exists,  based  on  the  offeror's  performance 
records,  that  the  offeror  can  perform  the  proposed  effort. 


Low 

Little  doubt  exists,  based  on  the  offeror's  performance 
records,  that  the  offeror  can  perform  the  proposed  effort. 

N/A 

No  performance  record  identifiable. 
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Chapter  5  Incorporating  SCE  into  the 

Relevant  Acquisition 
Documentation 

This  chapter  presents  the  major  documents  related  to  the 
source  selection  process  affected  by  the  incorporation  of  SCE. 
The  documents  examined  in  this  chapter  are:  Commerce 
Business  Daily  (CBD)  announcement.  Source  Selection  Plan 
(SSP),  Pre-proposal  Conference  presentation,  RFP,  and  the 
Evaluation  Standards.  A  discussion  accompanies  each  section 
describing  why  a  particular  approach  was  taken.  Each  section 
contains  at  least  one  example  that  can  be  tailored  to  the 
unique  needs  of  the  acquisition. 

Figure  5-1  presents  a  slightly  modified  SCE-related  portion  of 
an  actual  CBD  announcement.  This  annovmcement  informed 
the  potential  offerors  that 

•  SCE  would  be  performed  to  identify  the  offeror's  software 
process  capability. 

•  An  offeror's  software  process  capability  would  be  an 
integral  part  of  the  source  selection  decision. 

•  The  government  was  linking  quality,  cost,  and  schedule 
performance  directly  to  software  process  capability. 

•  The  offeror  should  have  a  SPEP  in  place  designed  to 
achieve  higher  maturity  levels  of  the  CMM. 

The  message  from  the  government  is  that  offerors  should  be 
implementing  process  improvement  programs  to  achieve 
higher  levels  of  process  capability  and  should  have  a  program 
in  place  to  be  a  "viable"  competitor.  The  RFP  that  would 


Making  the 
Commerce 
Business  Daily 
Announcement 
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follow  a  CBD  announcement  such  as  that  shown  in  Figure  5- 
1  could  reinforce  this  concept  by  requiring  the  submission  of 
a  SPIP  as  an  appendix  to  the  offerors'  proposals. 

The  statement  in  the  CBD,  "Contractors'  software  process 
capability  will  be  verified  by  analyzing  key  process  areas 
(KPAs)  through  the  Defined  maturity  level,  and  a 
demonstrable  software  process  improvement  program 
leading  to  levels  of  process  capability  higher  than  the  current 
capability"  makes  it  clear  that  the  Defined  maturity  level  is 
not  a  contract  requirement.  Rather,  it  is  a  standard  by  which 
the  evaluation  will  be  conducted,  and  the  source  selection  will 
consider.  It  essentially  defines  the  target  process  capability, 
which  is  the  capability  the  acquirer  is  seeking  for  this 
particular  acquisition  program. 


^  . . .  -  ■  -  _ *-  ^ . . . . 

As  part  of  <Acquisition  Agency's>  overall  effort  to  improve  software  quality,  cost,  and  schedule 

1 

performance,  the  software  process  capability  of  the  responding  offerors  will  be  a  consideration  in  the 
source  selection  decision.  <Acquisition  Agency>  will  use  the  Software  Capability  Evaluation  (SCE) 

.  .  : 

method  developed  by  the  Software  Engineering  Institute  (SEI)  to  evaluate  the  current  software 

. 

process  of  responding  offerors  (see  Capability  Maturity  Model  for  Software  (CMU/SEI-93-TR-24, 

February  1993).  Offerors  who  are  determined  to  be  in  the  competitive  range  wilt  have  their  software 

process  capability  verified  by  an  SCE  team.  The  SCE  team  will  analyze  key  process  areas  (KPAs) 

:  '  ' 

through  the  Defined  maturity  level  and  look  for  a  software  process  improvement  program  leading  to 

'  •• 

levels  of  process  capability  higher  than  the  current  capability.  A  conference  will  be  held  at  <time>  on 

si 

<date>  at  <where>  to  answer  any  questions. 

■>'  ; 

-• 

'  '  V  ‘  ,  -  - 

Figure  5-1  Sample  CBD  Announcement 


Why  place  SCE  wording  into  the  CBD  annoimcement?  SCE  is 
a  relatively  new  technique,  and  not  all  offerors  will  be  familiar 
with  it  or  the  government  use  of  SCE.  Describing  SCE  in  the 
CBD  allows  the  acquisition  agency  to  further  define 
requirements  so  industry  has  a  better  imderstanding  of  the 
requirements.  The  need  to  place  SCE  wording  into  the  CBD 
annoimcement  may  diminish  in  the  future  after  SCE  use  and 
process  improvement  activities  become  routine  business 
practices  throughout  the  acquisition  and  software 
development  communities. 
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Placing  SCE  in 
the  Source 
Selection  Plan 


Source  Screening 


The  Source  Selection  Plan  (SSP)  outlines  how  the  source 
selection  will  be  conducted  and  subsequently  how  the  award 
decision  will  be  made.  This  document  is  not  seen  by  the 
offerors.  SCE  has  a  relatively  small  impact  on  the  production 
of  the  SSP.  SCE  is  discussed  in  several  places  in  an  SSP.  The 
following  subsections  address  several  of  these. 

In  this  case,  the  government  would  publish  a  sources-sought 
synopsis  in  the  CBD  requesting  that  interested  offerors 
provide  to  the  government  their  qualifications  in  any  one  of  a 
number  of  technical  areas  important  to  the  acquisition.  The 
purpose  of  this  activity  is  to  produce  a  list  of  technically 
qualified  offerors.  Maturity  level  should  never  be  used  as  a 
screening  criterion  at  tins  stage.  However,  the  presence  of  an 
ongoing  software  process  improvement  program  could  be 
used  as  a  screening  criterion. 


Figure  5-2  provides  sample  wording  to  place  SCE  in  the 
Source  Screening  section  of  the  SSP.  The  hypothetical  source 
selection  is  using  Ada  Software,  Radar  Signal  Processing,  and 
Software  Engineering  Capability  as  screening  criteria.  The 
last  statement  in  this  example  communicates  the 
organization's  desire  to  keep  SPA  results  out  of  the  source 
selection  process.  The  SCE  team  should  not  ask  to  see  SPA 
results  when  on  site  (see  Chapter  1  for  detail  on  the 
differences  in  the  two  methods).  Many  organizations'  process 


improvement  programs  can  be  imdermined  by  offerors  trying 


to  demonstrate  to  the  government  a  process  capability  they 


cannot  support  on  a  new  program. 
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3.0 

3.1 


Source  Screening 

Screening  Criteria.  Initial  screening  of  potential  offerors  was  conducted  by  the  publication  of  a 
sources-sought  synopsis  in  the  Commerce  Business  Daily  on  <date>.  The  synopsis  requested 
that  interested  offerors  provide  their  qualifications  in  the  following  areas; 

a.  Software  Engineering  Capability.  Knowledge  of  software  process  improvement  with  a 
verifiable  program  for  process  improvement  using  the  Software  Capability  Evaluation  (SCE) 
Method  developed  by  the  Software  Engineering  Institute  (SEI)  to  evaluate  the  current  software 
process  of  responding  offerors  (Capability  Maturity  Model  (CMM)  (CMU/SEI'93-TR-24,  Feb 
93))  The  offerors  who  are  determined  to  be  in  the  competitive  range  after  initial  government 
evaluation  of  proposals  is  completed  will  have  their  software  process  capability  verified  by  an 
SCE  team.  The  SCE  team  will  evaluate  key  process  areas  (KPAs)  through  the  Defined 
maturity  level  and  look  for  a  demonstrat£d>le  software  process  improvement  program  leading 
to  levels  of  process  capability  higher  than  the  current  capability.  Do  not  provide  Software 
Process  Assessment  (SPA)  results. 


SCE  as  a  Specific 
Criterion 


Figure  5-2  SCE  as  a  Screening  Criterion  in  the  SSP 

The  following  example  shows  how  to  use  SCE  as  a  specific  criterion. 
Keep  in  mind  that  this  is  only  an  example.  Each  application  of  SCE 
should  be  tailored  to  accommodate  the  unique  demands  of  the 
acquisition. 

Figure  5-3  provides  a  detailed  description  of  how  a  source  selection 
could  use  SCE  in  the  source  selection  evaluation  process. 
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6.4  Evaluation  Criteria 

6.4.1  Assessment  Criteria 

a.  Soundness  of  approach 

b.  Compliance  with  requirements 

6.4.2  Specific  Criteria 

a.  Technical  Area  -  This  area  will  evaluate  three  items  (Software  Engineering  Capability  (SEC), 
Technical  (TECH)  approach,  and  Management)  represented  here  in  descending  order  of 
importance: 

1 .  Software  Engineering  Capability.  The  government  will  evaluate  the  software  process  by 
reviewing  the  offeror's  Software  Process  Improvement  Plan  (SPIP)  and  by  using  the 
Software  Engineering  Institute-  (SEI)  developed  Software  Capability  Evaluation  (SCE) 
Method.  The  government  will  determine  the  software  process  capability  by  investigating  the 
Key  Process  Area  (KPAs)  defined  in  the  SEI  Technical  Report,  Capability  Maturity  Model  for 
Software  (CMU/SEI-93-TR-24,  February  1993).  The  report  contains  a  description  of  the 
Capability  Maturity  Model  (CMM).  The  government  will  perform  an  SCE  of  each  offeror 
remaining  in  the  competitive  range  by  reviewing  current  projects  at  the  site  proposing  on  this 
contract  and  comparing  processes  used  on  those  projects  in  the  written  proposal/SPIP. 

The  evaluation  will  result  in  a  composite,  substantiated  through  individual  interviews  and 
reviews  of  documentation,  of  the  offeror's  software  process  on  the  government-selected 
projects.  A  risk  assessment  to  compare  proposed  practices  to  current,  vaiidated  practices 
may  be  performed.  The  evaluation  team  will  determine  findings  of  the  offeror’s  strengths, 
weaknesses,  and  improvement  activities  in  all  KPAs  through  the  Defined  maturity  level. 
Results  of  the  SCE  will  not  be  pass/faii.  Identified  weaknesses  will  be  provided  in  writing 
during  subsequent  discussions.  The  offeror  will  be  allowed  to  respond  to  the  findings  with  a 
limited  number  of  pages  of  text.  The  on-site  evaluators  may  be  separate  and  distinct  from  the 
proposal  evaluation  team  and  may  include  a  government  contracting  representative.  All 
evaluators  have  been  trained  in  the  SCE  Method. 


Figure  5-3  SCE  as  a  Specific  Criterion  For  The  SSP 


In  Figure  5-3,  the  use  of  SCE  as  a  specific  criterion  falls  under 
one  of  three  items  imder  the  technical  area  (SEC,  TECH 
Approach  and  Management)  in  this  case,  SCE  is  the  most 
important  technical  item.  The  SCE  findings  in  this  case  will 
form  the  basis  of  an  item  color  code  and  risk  assessment  and 
will  be  compared  to  an  evaluation  standard  based  on  the 
Defined  maturity  level.  The  SCE  is  not  pass/fail — that  is, 
being  less  than  Defined  maturity  level  will  not  exclude  an 
offeror  from  the  competitive  range.  In  this  example,  all 
offerors  within  the  competitive  range  will  experience  an  SCE 
site  visit  and  be  given  the  opportunity  to  respond  to  their 
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evaluated  software  process  weaknesses  through  PFNs,  CRs, 
and  DR.  Complete  SCE  findings  (strengths,  etc.)  should  be 
provided  to  each  unsuccessful  offeror  during  the  post-award 
debriefings  and  to  the  winning  contractor  at  the  post-award 
conference. 

What  is  presented  in  Figure  5-3  tracks  closely  with  what  is 
presented  in  a  later  section  of  this  chapter,  "Placing  SCE  in  the 
Request  for  Proposal." 

SCE  as  part  of  the  Figure  5-4  shows  an  example  for  using  SCE  findings  with  a 

PRAG.  Note  that  in  this  example  the  SCE  findings  are 
incorporated  with  other  factors  into  the  PRAG  analysis. 


6-3  Present  and  past  performance  (Performance  risk) 


(1 )  Present  and  past  performance  is  a  consideration  in  all  acquisition  agency  source 

selections.  Performance  risk  is  a  structured  treatment  of  present  and  past  performance 
and  will  be  determined  for  each  area.  It  is  a  confidence  measure  that  assesses  the 
offeror’s  present  and  past  work  record  in  order  to  determine  the  offeror's  ability  to  perform 
the  proposed  effort.  Performance  risk  is  assessed  by  the  Performance  Risk  Anaiysis  Group 
(PRAG),  which  should  be  chaired  by  a  program  manager-level  (senior)  individual  and 
consist  of  government  personnel.  Performance  evatuatbn  and  risk  assessment  focus  on 
relevant  past  performance  as  it  relates  to  the  specific  criteria.  It  is  the  PRAG’s 
responsibility  to  analyze  the  data  collected;  determine  its  relevance;  and  to  perform  an 
‘independenf  risk  assessment.  The  results  of  the  SCE  will  also  be  factored  into  the  overall 
performance  risk  rating  assigned  by  the  '^RAG.  For  information  on  how  to  establish  a 
PRAG,  how  it  operates,  the  forms  which  are  used,  and  how  the  evaluation  is  made  and 
reported,  refer  to  the  acquisition  agency  Source  Selection  Handbook. 


Figure  5-4  SCE  as  part  of  the  PRAG  for  SSP 
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Presenting  SCE 
at  the  Pre- 
Proposal 
Conference 


The  pre-proposal  conference  is  an  important  opportunity  for 
the  offerors  to  learn  the  specific,  detailed  requirements  of  the 
contract  and  for  the  government  to  communicate  its 
intentions  and  receive  feedback  from  the  potential  offerors. 
This  section  presents  a  modified  sample  briefing  used  during 
a  pre-proposal  conference  to  explain  the  SCE  process  and 
solicit  feedback  from  the  prospective  offerors.  Figure  5-5 
provides  a  title  slide  and  agenda.  The  presentation  consists  of 
the  following  parts  to  guide  the  interaction  with  the 
prospective  offerors: 

•  The  activities  that  usually  take  place  in  an  SCE  (Figure  5- 

6). 

•  The  SCE  team  process  (Figure  5-6). 

•  A  description  of  validation  procedures  (Figure  5-6), 

•  A  sample  of  the  documentation  that  may  be  looked  at  by 
the  SCE  team  (Figure  5-7), 

•  A  sample  site  visit  schedule,  (Figure  5-8). 

•  The  CMM  and  KPAs  against  which  the  team  will  evaluate 
the  offerors  (Figure  5-9), 

•  A  sample  of  the  findings  the  SCE  team  will  produce  before 
leaving  the  site  (Figure  5-10). 
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SCE  description 
SCE  team  activities 
Key  points  about  the  site  visit 
Criteria  for  validating  practices 
Sample  of  documents  reviewed 
Example  of  typical  site  visit  schedule 
Key  Process  Areas  (KPAs)  investigated 
Example  detailed  summary  findings 


USE  OF 

SOFTWARE  CAPABILITY  EVALUATION 
(SCE) 

IN  THE 

XYZ  CONTRACT 


Figure  5-5  Pre-proposai  Conference  Title  and  Agenda 
Slides 


Figure  5-6  shows  the  slides  used  to  describe  the  SCE  Method 
and  give  the  offeror  a  feel  for  the  site  visit  activities.  The  SCE 
Method  is  relatively  new  in  practice  and  there  are  offeror 
locations  that  have  not  experienced  an  SCE.  For  this  reason,  it 
is  especially  important  to  take  the  time  to  answer  any  SCE- 
related  questions  in  an  open  forum  such  as  an  offerors' 
conference.  Emphasis  should  be  placed  on  how  the  interviews 
and  document  reviews  will  be  conducted. 


One  of  th^  key  differences  between  the  SCE  and  SPA  methods 
is  the  examination  of  documentation  to  validate  the  processes. 
Keep  in  mind  when  presenting  the  slide  on  validating 
practices  that  validation  occurs  after  examining  the  audit  trail 
information.  This  audit  trail  information  reveak  how  certain 
processes  or  practices  are  implemented.  The  documents 
which  may  help  describe  these  practices  are  listed  in  Figure  5- 
7. 
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SCE  DESCRIPTION 

A  method  for  evaluating  the  software 
process  of  an  organization  to  gain  insight 
into  its  software  development  capability. 


SCE  TEAM  AC  TIVITIES 

Compare  RFP  product  profile  with  offeror 
project  profiles  to  select  up  to  four  projects 
for  investigation. 


mi 


Five  phase  evaluation  process  using  the 
SEI CMM  as  the  basis  for  evaluation. 

Three-day  site  visit  conducted  by  a  four  to 
six-person  evaluation  team. 

Outcome:  findings  on  process  capability  in 
terms  of  software  development  key  process 
areas  (KPAs) — strengths,  weaknesses,  and 
improvement  activities. 


Analyze  maturity  questionnaire  responses. 

Set  expectations  (in-brief)  and  receive 
process  presentation  from  contractor. 

Interview  management  and  key  personnel.  ^  ^ 

Analyze  substantiating  documentation. 

Produce  findings  through  consensus. 


No  recommendations  will  be  made. 


KEY  POINTS  ABOUT  THE  SCE  SITE  VISIT 


Team  decisions  made  through  consensus. 


CRITERIA  FOR  VALIDATING  PRACTICES 

•  Written  procedure  for  a  practice  exists. 


wmi 


I 


*3 
if'  ,'.1 


All  interviews  confidential. 

The  SCE  team  may  inten/iew  an  individual 
more  than  once  during  the  same  visit. 

Interview  schedule  will  become  dynamic 
after  the  first  day. 

Intenriew  process  works  down  the  chain  of 
command. 

Working-level  development  personnel  will  be 
interviewed. 


Procedure  implementing  practice  must  be 
effective  (work  for  the  organization). 

Evidence  exists  showing  track  record  that 
procedures  are  followed. 

Practice  implemented  for  at  least  one  year 
(those  less  than  one  year  may  be  listed  as 
improvement  activities). 


1.  4 


Document  review  occurs  throughout 
process. 

SCE  team  will  not  make  recommendations. 


Figure  5-6  Pre-proposal  Conference  SCE  Method  Slides 


CMU/SEI-94-TR-5 


5-93 


Chapter  5:  lrKH>rporating  SCE  into  the  Relevant  Acquisition  Documentation 


Figure  5-7  provides  a  broad  list  of  the  documentation  that  the 
SCE  team  may  examine  depending  upon  the  course  of  the 
interviews.  This  list  of  documents  is  a  guide  to  better  prepare 
the  offerors.  Many  more  documents  may  actually  be  reviewed 
during  the  site  visit.  Offerors  should  be  encouraged  to  obtain 
the  Software  Capability  Evaluation  Version  2.0  Method 
Description  (SCE  94]  to  learn  more  about  the  SCE  Method. 


Figure  5-7  Pre-proposal  Conference  Documentation  Slide 
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The  site-visit  schedule  (Figure  5-8)  should  be  mentioned 
during  the  briefing  in  order  to  prepare  offerors  for  what  will 
occur  during  the  site  visit.  It  should  be  emphasized  that  the 
SCE  team  will  do  everything  possible  to  minimize  the  impact 
on  the  offeror's  software  development  organization. 

If  the  team  is  planning  to  visit  the  engineering  floor  (software 
engineer's  work  areas)  at  each  site,  the  team  should  explain 
how  this  will  be  done,  should  explain  that  the  SCE  team  will 
do  this  activity  at  a  mutually  agreeable  time  during  the  site 
visit,  and  should  estimate  the  duration  of  the  floor  visit 
(normally  NTE  4  hours  per  project  visited). 
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SAMPLE  SITE  VISIT  SCHEDULE 


DAYO 

0800-1330 

1330-1800 

DAY1 

0800-0830 

0830-1030 

1030-1230 

1230-1330 

1330-1430 

1430-1600 

1600-1730 

1730-1800 

DAY  2 

0800-1000 

1000-1200 

1200-1300 

1300-1400 


Travel  for  SCE  team 
Prepare  for  she  vish 


Contractor  welcome/introduction  and  SCE  team’s  orientation  briefing 
Contractor  software  presentation/contractor  understanding  caucus 
Initial  review  of  organizational  documents 
Lunch 

SCE  team  inten/iews  -  senior  managers 
SCE  team  inten/iews  -  program  managers 
SCE  team  interviews  -  software  managers 
SCE  team  caucus  and  requests  documents 


SCE  team  interviews  continued  whh  program  managers,  software 
managers 

SCE  team  intenriews  •  managers  (engineering,  SQA,  SCM,  test,  SEPQ) 
Lunch 

SCE  team  caucus,  request  and  review  documents 


DAYS 

0800-1000 

1000-1200 

1200-1300 

1300-1500 


SCE  team  inten/iews  •  key  personnel  (may  include  engineering  staff) 
Review  addhional  documentation 
Lunch 

Additional  documentation  review  and/or  addhional  SCE  team  interviews  • 
Consolidation  interviews  with  managers,  engineers,  and  staff 
1 500-1800  SCE  team  caucus  and  preparation  of  findings 


DAY  4 

0800-0900 

0900-1000 

1000-1100 

1100-1730 


Final  preparation  of  findings 
Exit  meeting  whh  offeror 
SCE  team  caucus  and  wrap-up 
Travel  for  SCE  team 


Figure  5-8  Pre-proposal  Conference  Site  Visit  Schedule 
Slide 


The  CMM  should  also  be  explained  so  that  the  basis  of  the 
SCE  is  clear  (Figure  5-9).  The  offerors  need  to  understand 
what  KPAs  are  going  to  form  the  basis  of  the  specific  criterion, 
or  be  considered  under  performance  risk,  incorporating  the 
SCE  findings  into  the  source  selection  decision.  Depending 
upon  the  offeror  pool  and  their  familiarity  with  the  CMM, 
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additional  slides  may  be  needed  to  explain  the  subprocess 
areas  or  key  practices  for  each  KPA  the  team  will  be 
examining  during  the  SCE. 


Level 

Characteristic 

Key  Process  Area 

Optimizing  (5) 

•  Continuous  process 
improvement  capability 

•  Process  change  management 

•  Technology  innovation 

•  Defect  prevention 

Managed  (4) 

•  Product  quality  planning 
and  tracking  of 
measured  software 
process 

•  Process  measurement  and 
analysis 

•  Quality  management 

Defined  (3) 

•  Life  cycle  process 
defined  and 
institutionalized  to 
provide  product  quality 
control 

•  r'eer  Reviews 

•  Intergroup  coordination 

•  Software  product  engineering 

•  Integrated  software 
management 

•  Training  program 

•  Organization  process 
definition 

•  Organization  process  focus 

Repeatable  (2) 

•  Management  oversight 
and  tracking  of  project 

•  Stable  planning 

•  Software  configuration 
management 

•  Software  quality  assurance 

•  Software  subcontract 
management 

•  Software  project  tracking  and 
oversight 

•  Software  project  planning 

•  Requirements  management 

Initial  (1) 

•  Ad  hoc  (unpredictable, 
chaotic) 

Figure  5-9  Pre-proposal  Conference  CMM  Slide 

The  last  slide  (Figure  5-10)  in  the  SCE  portion  of  the  offerors' 
conference  briefing  explains  to  the  offerors  what  the  results  of 
the  site  visit  will  look  like  and  how  they  will  be  presented  to 
the  SSEB.  It  is  not  possible  to  give  specifics  on  the  evaluation 
standard  or  the  percentage  that  the  technical  item  Software 
Engineering  Capability  contributes  to  the  source  selection 
decision;  however,  the  order  of  importance  SCE  has  v  en 
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used  as  an  item  imder  the  technical  area  can  be  emphasized. 
This  is  an  opportunity  to  reinforce  the  point  that  all  findings 
are  made  through  consensus  by  the  team.  Sample  findings 
charts  should  reflect  the  key  process  areas  that  are  going  to  be 
evaluated  for  the  acquisition. 


SAMPLE  DETAILED  RNDING 
SQA  KPA 

Strengths 

•  Independent  reporting  chain 

•  Highly  visible 

•  Insuring  software  engineering  standards 
compliance 

•  Management  commitment  -  strong  staff 

Weaknesses 

•  Inconsistent  auditing 

•  Ineffective  use  of  resources 


Improvement  Activities 

•  Establishing  procedures  for  consistent 
auditing 


EXAMPLE  SUMMARY  FINDING 
Strong 

•  Software  Quality  Assurance  (SQA) 


Acceptable 

•  Software  Project  Planning 

•  Software  Configuration  Management 

•  Requirement  Management 

Weak 

•  Software  Subcontract  Management 

•  Organization  Process  Definition 

•  Training  Program 

•  Peer  Reviews 


Figure  5-10  Pre-Proposal  Conference  Sample  Findings 
Slides 


Placing  SCE  in 
the  Request  for 
Proposal 


•  Evaluation  criteria. 

•  The  mechanisms  that  will  be  employed  in  aking  the 
source  selection  decision. 

•  How  to  propose  for  the  contract. 


This  section  contains  the  information  needed  to  incorporate 
SCE  into  the  RFP.  The  REP  is  the  document  used  by  the 
government  to  explain  to  offerors 

•  The  government's  requirements. 
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General  Language 
for  the  Request  for 
Proposal 


All  of  the  examples  in  this  section  revolve  around  Software 
Engineering  Capability  being  used  as  a  specific  criterion  in 
the  Technical  area  of  the  proposal.  If  the  SCE  findings  are 
planned  to  be  used  as  a  consideration  under  performance 
risk,  these  examples  can  be  easily  tailored  to  meet  such  usage. 

Regardless  of  how  SCE  is  to  be  used  in  making  the  source 
selection  decision,  the  description  of  its  use  is  found  in  Section 
L  (Instructions,  Conditions,  and  Notices  to  Offerors)  and  M 
(Evaluation  Factors  for  Award)  of  the  RFP.  The  example 
shown  in  Figure  5-11  closely  mirrors  an  actual  description  of 
SCE  use  found  in  Section  M  of  an  RFP.  The  example  begins  by 
identif5dng  Software  Engineering  Capability  as  an  item  imder 
the  technical  area  (specific  criterion). 

For  this  example,  there  were  two  areas  of  evaluation:  1) 
Technical  and  2)  Cost.  The  specific  reference  to  SCE  in  the  RFP 
begins  by  describing  in  general  terms 

•  What  will  be  evaluated  in  the  technical  area  (process 
capability)  and  the  importance  placed  on  each. 

•  What  the  technical  basis  of  the  evaluation  is  (the  CMM 
KPAs) 

•  What  the  results  of  the  evaluation  will  be  (identify 
strengths,  weaknesses,  and  risk  which  will  also  consider 
improvement  activities  by  KPA) 

•  How  the  government  will  conduct  the  evaluation  (select 
the  projects  to  be  reviewed,  conduct  interviews,  and 
review  documentation). 
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B.  Evaluation  Criteria 


1.0  Introduction 


2.0  Basis  for  Contract  Award 


3.0  General  Considerations 


4.0  Assessment  Criteria 


4.1  Specific  Criteria 


4.1.1  Technical  Area 

a.  Technical  Area  -  This  area  will  evaluate  three  items  (SEC,  TECH  approach  and 
Management)  represented  here  in  descending  order  of  importance: 

1 .  Software  Engineering  Capability.  The  government  will  evaluate  the  software  process  by 
reviewing  the  offeror’s  Software  Process  Improvement  Plan  (SPIP)  and  by  using  the 
Software  Engineering  Institute  (SEI)-developed  Software  Capability  Evaluation  (SCE).  The 
government  will  determine  the  software  process  capability  by  investigating  the  Key 
Process  Area  (KPAs)  defined  in  the  SEI  Technical  Report,  Capability  Maturity  Model  for 
Software  (CMU/SEI-93-TR-24).  The  report  contains  a  description  of  the  Capability  Maturity 
Model  (CMM)  and  the  SEI-defined  maturity  levels.  The  government  will  perfoim  an  SCE  of 
each  offeror  remaining  in  the  competitive  range  by  reviewing  current  projects  at  the  site 
proposing  on  this  contract  and  comparing  methods/processes  used  on  those  projects  in 
the  written  proposal/SPIP. 

The  evaluation  will  result  in  an  organizational  composite,  substantiated  through  individual 
interviews  and  reviews  of  documentation,  of  the  offeror’s  software  process  activities  on  the 
government-selected  projects.  A  risk  assessment  to  compare  proposed  practices  to 
current,  validated  practices  may  be  performed.  ITie  evaluation  team  will  determine  findings 
of  the  offeror’s  strengths,  weaknesses,  and  improvement  activities  in  all  KPAs  through  the 
Defined  maturity  level.  Results  of  the  SCE  will  not  be  pass/fail.  Identified  weaknesses  will 
be  provided  in  writing  during  subsequent  discussions.  The  offeror  will  be  allowed  to 
respond  to  the  findings  with  a  limited  number  of  pages  of  text.  The  on-site  evaluators  may 
be  separate  and  distinct  from  the  proposal  evaluation  team  and  may  include  a  government 
contracting  representative.  All  evaluators  have  been  trained  in  the  SCE  Method. 

4.2  Cost/Price  Area... 


Figure  5-1 1  General  RFP  Language  To  include  SCE  (RFP 
Section  M) 


Eliminating 
References  to 
Maturity  Levels  in 
the  RFP 


While  Figure  5-11  treats  the  maturity  level  as  a  basis  for 
evaluation  rather  than  a  requirement,  it  is  recommended  that 
references  to  maturity  level  be  eliminated  to  the  greatest 
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References  to  SCE 
in  the  Instructions 
for  Proposal 
Preparation 


extent  possible.  One  way  to  do  this  is  by  listing  the  actual 
KPAs  and  subprocess  areas  or  key  practices  to  be  evaluated 
directly  in  the  RFP  rather  than  simply  referencing  the  KPAs 
found  in  the  CMM.  Eliminating  references  to  maturity  levels 
allows  the  acquirer  to  establish  a  target  process  capability 
using  the  CMM  KPAs  and  also  to  be  specific  as  to  the  basis  of 
evaluation.  It  is  the  findings  against  the  KPAs  tiiat  are  most 
important  to  the  acquirer  in  determining  specific  risk  in  an 
offeror,  not  the  maturity  level. 

The  IFPP  portion  of  the  RFP  informs  the  offerors  how  to 
prepare  their  proposals  and  comply  with  the  source  selection 
process.  The  portion  of  the  IFPP  that  addresses  SCE  is  divided 
into  five  distinct  pieces: 

1.  Organizing  the  SCE-related  information  into  a  separate 
appendix. 

2.  Completing  die  Assessment  Recording  Forms  (Figure  5- 

12). 

3.  Completing  the  Project  Profiles  (Figure  5-13). 

4.  Submitting  the  Software  Process  Improvement  Plan  (SPIP) 
(Figure  5-14). 

5.  General  SCE  instructions  (Figure  5-15). 

Organizing  SCE-related  Information  into  an  Appendix 
Each  acquisition  must  determine  the  best  way  for  their 
program  to  instruct  offerors  to  organize  their  proposals.  The 
goal  is  to  ease  proposal  evaluations  and  obtain  concise 
information  wanted,  and  not  to  impose  a  burden  on  the 
offerors.  Work  with  your  PCO  to  determine  what  is  best  for 
your  acquisition  cmd  program.  If  it  is  desired  that  the  SCE 
information  be  excluded  from  the  technical  page  limit,  you 
may  want  offerors  to  place  this  information  in  a  separate 
appendix. 
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Completing  Assessment  Recording  Forms 
One  of  the  more  important  SCE-related  itenw  in  the  IFPP  is 
the  language  shown  in  Figure  5-12  describing  how  the 
Assessment  Recording  Forms  are  to  be  completed.  The  offeror 
is  told  to  select  eight  ongoing  development  efforts  that  best 
correlate  to  the  future  work  tmder  the  govenunent's 
proposed  contract,  using  characteristics  such  as  application 
domain,  software  size,  development  language,  etc.  to  make 
the  best  matches. 
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irmmi 


ropoMi 


3. 1  General  Instructions  The  Technical  Proposal  shall  consist  of  the  Executive  Summary  and 
<n>  separate  sections... 

3.2  Executive  Summary... 

3.3  Specific  Instructions... 

3.3.1  Section  I  -  Software  Engineering  Capability.  The  offeror  will  provide  the  following 

information  to  assist  the  government's  preparation  for  the  Software  Capability  Evaluation  of 
each  offeror 

a.  The  offeror  will  complete  the  SEI  Maturity  Questionnaire  (MQ)  (current  version)  using  the 
Assessment  Recording  Form  for  eight  current  projects  at  the  site  proposing  on  this 
solicitation  (a  project  is  acceptable  only  if  it  has  been  completed  in  the  last  year).  The 
offeror  should  select  those  projects  that  best  match  the  engineering  requirements  of  this 
contract.  For  offerors  with  fewer  than  eight  currant  projects  at  the  proposing  site,  submit 
MQ  responses  for  as  many  projects  as  are  available.  For  each  ‘Ves’  response,  please 
note  on  the  comment  line  the  mechanism  or  documentation  justifying  the  response.  The 
MQ  can  be  found  in  Atch  XXX  of  the  IFPP.  The  completed  Assessment  Recording  Forms 
will  be  submitted  with  the  proposal.  For  ail  responses,  please  note  at  the  start  of  the 
comment  line  the  degree  of  implementation  of  each  practice  using  a  letter  identifier  from 
the  following  legend: 

A  -  Not  implemented  at  this  time. 

B  •  Not  implemented  at  this  time,  but  desired. 

C  -  Currently  planning  to  implement.  See  improvement  plan. 

D  •  In  the  process  of  implementing. 

E  •  Implemented  with  less  than  a  year's  experience. 

F  -  Implemented  on  a  project-by-project  basis. 

G  -  Implemented  organizationally. 

H  -  Not  appropriate  for  our  organization. 


Figure  5-12  instructions  For  Compieting  Assessment 
Recording  Forms 

Using  the  legend  in  Figure  5-12,  the  offeror  must  characterize 
the  state  of  institutionalization  of  each  practice.  To  verify  each 
response,  the  offeror  mvist  cite  documentation  that  defines 
each  practice  it  has  characterized.  By  getting  this  information 
from  offerors,  the  SCE  team  will  know  more  about  what  to 
look  for  to  verify  a  particular  software  practice,  and  the 
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offeror's  view  of  the  extent  to  which  a  practice  is 
implemented;  this  will  help  focus  on-site  activities  (e.g., 
interviews). 

The  SCE  team  wants  to  identify  and  separate  software 
practices  that  are  institutionalized  (implemented 
organizationally)  from  those  that  are  implemented  only  on  a 
single  project.  The  last  reference  in  Figure  5-12  is  to  the 
maturity  questionnaires  the  offeror  must  complete  for  each 
project  submitted  in  order  to  satisfy  this  request.  The 
questionnaire  should  be  attached  to  the  RFP  so  that  there  is  no 
confusion  over  which  questionnaire  the  offeror  is  required  to 
complete  (A  Method  far  Assessing  the  Software  Engineering 
Capabilty  of  Contractors  (Humphrey  87b). 

Completing  the  Project  Profiles 

Figure  5-13  directs  the  offeror  to  complete  a  Project  Profile  for 
each  of  the  projects  selected  on  the  Assessment  Recording 
Form.  The  project  profile  provides  information  such  as:  size 
of  the  organization,  application  area,  development  language, 
type  of  system,  location  of  development,  and  the  program's 
current  phase(s)  of  development.  This  information  helps  the 
government  in  selecting  projects  for  evaluation.  A  Project 
Profile  should  also  be  included  as  an  attachment  to  the  IFPP. 


(continued) 


b.  For  each  project,  the  offeror  will  complete  a  Project  Profile,  attach  it  to  the  respective 
Assessment  Recording  Form,  and  submit  it  with  the  proposal.  The  Project  Profile  template 
can  also  be  found  in  Atch  XXX  of  the  IFPP.  This  document  shall  be  no  greater 
than  one  page  per  project. 


Figure  5-13  Instructions  For  Completing  Project  Profiles 

Preparing  the  Software  Process  Improvement  Plan 
Figure  5-14  addresses  the  SPIP  the  offerors  submit  with  their 
proposak.  In  the  example  provided  the  offeror  could  not 
exceed  15  pages  of  text.  This  was  an  arbitrary  limit  intended 
to  minimize  the  effort  required  by  the  offerors  and  the 
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government  during  the  source  selection  process.  The 
government  did  not  want  the  offerors  to  develop  an  SPIP  for 
tb.e  acquisition;  rather,  they  wanted  to  see  plans  that  were 
already  in  place. 


3.31  (continued) 


c.  The  offeror  will  submit  the  proposing  site's  Software  Process  Improvement  Plan  (SPIP), 
in  the  format  of  their  choosing,  with  the  proposal.  The  document  will  be  no  more  than  15 
pages.  The  SPIP  should  communicate  the  offeror’s  current  software  process  capability  as 
well  as  their  desired  maturity  level,  specific  planned  improvements,  dedicated  resources, 
effort  estimates,  and  a  time  phasing  of  those  improvements  to  bring  the  offeror's  software 
process  capability  to  the  organization's  desired  maturity  level. 


Figure  5-14  Instructions  For  Submitting  the  Software 
Process  Improvement  Plan  (SPIP) 


General  SCE  Instructions 

The  last  set  of  instructions  for  the  IFPP,  foimd  in  Figure  5-15, 
informs  the  offeror  of  various  SCE-related  details  that  will 
facilitate  a  smoothly  run  SCE  with  minimal  impact  on  the 
offeror's  organization. 

•  An  Offeror  Point  of  Contact  is  needed  so  that  the  SCE  team 
leader  may  coordinate  all  activities,  both  before  and 
during  the  SCE.  Note  that  the  offeror  will  be  notified  five 
working  days  in  advance  of  the  site  visit  which  projects 
will  be  evaluated.  There  are  two  reasons  for  this.  First,  this 
will  give  all  the  offerors  the  same  number  of  days  to 
prepare  for  the  SCE.  Second,  because  many  organizations 
go  to  great  lengths  to  prepare  for  an  SCE,  giving  five 
working  days'  notice  will  limit  them  from  expending 
valuable  resources  and  time  on  activities  that  will  have 
little  or  no  impact  on  the  SCE  findings. 
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3.31  (continued) 

d.  After  the  proposal  is  received,  the  government  will  coordinate  a  site  visit  with  those 

* 

offerors  remaining  in  the  competitive  range  to  conduct  the  Software  Capability  Evaluation 

•  ■  -I 

(SCE)  at  the  offeror’s  location.  The  offeror  wilt  provide,  with  your  proposal,  a  point  of 

|||3 

contact  and  phone  number  at  the  offeror's  site  for  the  SCE  team  leader  to  coordinate  all 

SCE  activities.  The  government  will  also  communicate  details  about  the  site  visit  during 

the  coordination  process.  The  offeror  will  be  ratified  of  the  projects  to  be  examined 

approximately  five  working  days  prior  to  the  site  visit.  The  site  visit  dates  selected  by  the 

government  are  not  open  for  discussion. 

i 

e.  If  a  site  visit  is  conducted  with  your  firm,  the  SCE  team  will  need  a  closed  meeting  room 

; 

capable  of  accommodating  at  least  eight  people.  The  offeror  should  have  a  copy  of  the 

organization’s  software  standards,  procedures  and/or  operating  instructions,  and 

organizational  charts  for  the  projects  being  reviewed  in  the  meeting  room  when  the  SCE 

‘.J 

team  arrives.  All  interviews  conducted  as  part  of  the  SCE  will  be  done  in  private,  one 

individual  at  a  time. 

♦ 

f.  The  Assessment  Recording  Forms,  Project  Profiles,  and  Software  Process  Improvement 

Plans  will  not  be  included  in  the  page  count  limitations  for  the  proposal. 

Figure  5-1 S  Instructions  For  Site  Visit  Coordination 

•  Facilities  and  Information.  The  items  needed  by  the  team  at 
the  site  visit  are  mentioned  in  this  section.  This 
information  needs  to  be  provided  here  to  set  expectations 
and  ensure  that  the  offeror  is  reasonably  well  prepared. 
Note  that  private  interview  notes  will  always  remain, 
source  selection  sensitive.  The  SCE  team  must  maintain 
the  confidentiality  of  interviews  or  the  entire  SCE  process 
could  be  imdermined.  All  data  collected  during  the  site 
visit  will  become  part  of  the  source  selection  file  and  will 
be  maintained  on  all  offerors  until  the  contract  is  closed 
out. 

•  Offeror  Exit  Briefing.  The  PCO  will  be  the  final  arbiter  in 
determining  how  the  findings  will  be  provided  to  the 
offerors.  However,  any  outbriefing  must  advise  the  offeror 
that  this  may  not  completely  resolve  all  issues  regarding 
the  SCE.  It  is  important  for  all  the  offerors  to  realize  that 
they  have  the  right  and  must  specifically  request  a 
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debriefing  of  the  SCE  findings.  Debriefing  the  findings 
achieves  two  important  goals.  First,  in  a  Total  Quality 
Management  (TQM)  approach,  the  government  desires 
buy-in  from  the  offerors  regarding  the  results,  and  is 
seeking  to  motivate  the  offerors  to  improve  their 
capability.  Second,  the  government  has  the  opportunity 
for  direct  feedback  regarding  the  conduct  of  the  SCE  from 
the  offeror's  perspective.  This  feedback  can  be  used  to 
refine  the  procedures  and  instructions  for  future 
acquisitions. 

•  Page  Limitations.  In  most  RFPs,  there  is  a  limit  to  the 
number  of  pages  an  offeror  may  use  in  the  preparation  of 
their  proposal.  The  example  provided  here  had  such  a 
requirement.  Consequently,  when  the  IFPP  required 
submittal  of  Assessment  Recording  Forms,  Project 
Profiles,  and  an  SPIP,  these  dociunent  pages  were 
excluded  from  the  proposal  page  count  to  ensure  they  did 
not  detract  from  the  technical  content  of  the  proposal 
subject  to  the  page  limitations.  This  is  an  administrative 
detail  which  will  allow  page  coimts  to  be  focused  on  the 
technical  approach. 

This  section  presented  the  essential  elements  needed  to 
accommodate  SCE  in  an  RFP.  These  references  should  be 
tailored  by  the  organization  to  meet  the  specific  needs  of  the 
acquisition.  The  references  presented  can  be  changed  to 
accommodate  the  usage  of  the  SCE  findings  as  a 
consideration  under  performance  risk  or  a  variation  of  the 
specific  criterion  example  presented  here. 


Placing  SCE  in 
the  Evaluation 
Plan 


This  section  provides  a  sample  of  the  type  of  information 
needed  to  incorporate  SCE  into  the  Evaluation  Plan,  and  to 
assist  in  the  preparation  of  an  evaluation  standard  for  SCE 
findings. 
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Sample  Evaluation 
Standard 


Evaluation  standards  must  be  completed  before  RFP  release. 
Evaluation  standards  are  written  in  a  detailed  manner  to 
promote  competition  and  enhance  the  discrimination 
between  the  offerors. 

It  is  imperative  that  the  SCE  team  leader  be  a  member  or 
advisor  to  the  Source  Selection  Evaluation  Board  (SSEB)  to 
work  with  SSEB  members  to  apply  the  evaluation  standard  to 
the  findings  from  each  of  the  offerors. 

The  example  presented  in  Figure  5-16  is  a  sample  evaluation 
standard.  A  detailed  evaluation  standard  is  written 
describing  the  requirements  for  the  acquisition.  This  implies 
that  if  the  requirement  is  met,  the  standard  is  met,  and  the 
offeror  is  scored  Green.  If  the  standard  is  exceeded,  the  offeror 
is  scored  Blue.  If  the  requirement  is  not  met,  depending  on 
how  near  it  is  to  being  met,  th“  offeror  is  scored  as  Yellow.  A 
Red  score  denotes  serious  deficiencies  (failure  to  meet 
requirements).  Application  of  the  color  ratings  to  a  standard 
should  be  correlated  with  the  appropriate  regulations  and 
procurement  policies  affecting  your  acquisition. 
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1  DESCRIPTION:  Software  Engineering  Capability— The  government  will  evaluate  the  offeror's 
organizational  software  process  capability  by: 

Performing  a  Software  Capability  Evaluation  (SCE). 

Evaluating  the  offeror’s  program  for  software  process  improvement. 

Evaluating  the  extent  to  which  the  offeror’s  software  process  supports  the  goals,  objectives,  and 
requirements  of  the  solicitation. 

{STANDARD-  The  standard  Is  met  when  the  offeror  presents  a  sound,  compliant  approach  and: 

1 .  The  findings  from  the  SCE  show  that  the  offeror  is  strong  or  acceptable  in  each  of  the  following  key 
process  areas: 

•  Software  Configuration  Management 
•  Software  Quality  Assurance 
•  Software  Subcontract  management 
•  Software  project  tracking  and  oversight 
•  Software  project  planning 
•  Requirements  Management 

12.  The  findings  from  the  SCE  show  that  the  offeror  is  strong  or  acceptable  in  at  least  one  of  the  following  j 
software  process  areas: 

•  Peer  Reviews 
•  Intergroup  coordination 
•  Software  product  engineering 
•  Integrated  software  management 
•  Training  program 
•  Organization  process  definition 
•  Organization  process  focus 

13.  The  Software  Process  Improvement  Plan  submitted  with  the  offeror’s  proposal  portrays  the  offeror’s 
current  process  capability  realistically  and  presents  a  realistic  plan  for  process  improvement.  The 
offeror’s  plan  is  consistent  with  the  SCE  findings.  The  SPIP  outlines  the  offeror’s  plan  to  achieve 
higher  maturity  levels  and  demonstrates  that  the  offeror  understands  software  process  improvement, 
both  technically  and  in  the  effort  required  to  increase  and  sustain  higher  maturity  levels. 


Figure  5-16  Streamlined  SCE  Evaluation  Standard 

This  section  presented  an  example  of  an  evaluation  standard 
which  de-emphasizes  maturity  levels  while  keeping  witii  the 
spirit  of  the  CMM.  Trained  SCE  users  are  able  to  take  this 
example  and  tailor  it  to  meet  the  specific  needs  of  their 
acquisition.  Thus,  SCE  can  contribute  effectively  to  the  source 
selection  decision.  The  findings,  provided  to  the  SSEB  by  the 
SCE  team,  are  a  snapshot  of  process  capability  for  a  specific 
site  at  a  particular  point  in  time.  The  way  those  findings  are 
used  by  the  acquisition  organization  can  be  modified  through 
the  design  of  the  evaluation  standard. 
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Appendix  A  SCE  Participants  In  a  Source 

Selection 


Figure  A-1  shows  how  a  source  selection  organization  may  be 
organized  with  the  incorporation  of  SCE. 


Figure  A>1  Sample  Source  Selection  Organization 

Note  that  in  this  example  the  cost  team  and  contract  definition 
team  are  separate.  They  would  be  combined  when  the  SSET 
structure  is  used. 

The  following  are  the  principal  source  selection  players  who 
influence  the  SCE  implementation  decision,  whether  it  will  be 
used  and  to  what  extent  the  SCE  findings  will  play  a  role  in 
the  source  selection  decision. 
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Source  Selection  Authority  (SSA):  As  the  individual 
ultimately  responsible  for  the  conduct  of  the  source  selection 
organization,  the  SSA  is  the  final  arbiter  on  the  use  of  SCE  and 
approves  how  the  findings  will  influence  the  source  selection 
decision  and  how  they  will  be  disclosed  to  offerors. 

Source  Selection  Advisory  Council  (SSAC):  The  SSAC  is 
charged  with  collecting  and  analyzing  the  SSEB's  evaluations 
of  each  offeror.  This  is  the  only  group  permitted  to  compare 
the  strengths  and  weaknesses  of  the  offerors  against  one 
another.  The  SSAC  may  recommend  to  the  SSA  how  the  SCE 
findings  will  be  incorporated  into  the  source  selection 
decision  at  the  pre-RFP  release  briefing. 

SSAC  Chairperson:  This  individual  is  usually  the  facilitator 
of  the  entire  process  and  acts  as  the  SSA's  eyes  and  ears 
during  the  course  of  ti\e  source  selection.  This  person  is 
normally  the  2-LTR  deputy,  or  at  least  in  the  chain-of- 
command  above  the  program  manager.  This  individual  will 
coordinate  all  activities  with  the  SSA  and  facilitate  the 
consensus-building  process  from  within  the  SSAC. 

Performance  Risk  Analysis  Group  (FRAG):  The  PRAG 
collects  data  on  each  offeror's  past  and  present  performance 
on  other,  similar  product  acquisitions  and  from  this  data 
determines  relevance  of  effort  and  assesses  the  performance 
risks  posed  in  the  offeror's  ability  to  perform  if  the  offeror  is 
selected.  SCE  findings  can  be  factored  into  this  performance 
risk  assessment.  The  PRAG  does  more  than  assess  past 
performance.  It  may  also  assess  such  things  as  manufacturing 
capability,  plant  capacity,  or  an  offeror's  familiarity  with  the 
type  of  work  required  imder  the  instant  solicitation,  in  an 
attempt  to  analyze  an  offeror's  performance  risk.  The  PRAG 
normally  reports  directly  to  the  SSA  or  the  SSAC.  The  PRAG 
role  is  fully  defined  in  the  AFMC  FAR  supplement  to  AFR  70- 
15  and  AFR  70-30.  The  PRAG  needs  to  be  well-educated  on 
what  SCE  is  and  what  it  can  provide  so  that  the  right 
information  is  passed  to  the  SSAC.  In  these  instances,  at  least 
one  member  of  the  PRAG  should  be  an  SCE-trained 
government  individual. 
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Source  Selection  Evaluation  Board  (SSEB):  This  is  the 
government  group  that  evaluates  the  offerors'  proposals 
against  evaluation  standards  and  section  M  of  the  RFP.  This 
group  develops  the  evaluation  standards  and  receives 
approval  to  use  them  from  the  SSAC  and  SSA  before  the 
issuance  of  the  RFP.  The  SSEB  is  usually  organized  into 
technical  and  cost  teams  representing  each  of  the  evaluation 
areas  or  technical  disciplines  important  to  the  award  decision. 
Within  each  team,  each  individual  brings  the  technical  skills 
and  professional  experience  necessary  to  perform  a  fair  and 
effective  evaluation.  If  the  findings  of  an  SCE  are  being 
factored  into  the  source  selection  decision  as  an  Evaluation 
Criterion,  the  SCE  team  leader  should  be  a  member  of  the 
SSEB.  The  SSEB  prepares,  prior  to  the  release  of  the  RFP,  an 
evaluation  standard  that  will  incorporate  the  SCE  findings. 

SSEB/SSET  Chairperson:  The  SSEB/SSET  chairperson 
coordinates  all  activities  of  the  SSEB/SSET  related  to  the 
acquisition.  The  chairperson  will  facilitate  the  incorporation 
of  SCE  into  the  source  selection  documentation  and  monitor 
the  various  evaluation  teams,  including  the  SCE  team. 

Program  Director  /  Manager  The  program  director /manager 
normally  orchestrates  the  acquisition  from  the  vantage  point 
of  the  SSEB/SSET  chairperson,  depending  upon  the  nature  of 
the  organization  and  dollar  size  and  criticality  of  the  contract. 
The  SCE  team  leader  will  have  to  work  with  the  program 
director  (if  he  or  she  is  part  of  the  source  selection,  otherwise 
it  would  be  the  SSEB/SSET  chairperson)  during  the  source 
selection  to  perform  SCEs  as  part  of  the  proposal  evaluation 
process  and  place  them  in  the  source  selection  plan. 

A  program  director  of  a  large  program  may  have  a  number  of 
contracts  imder  one  program  that  he  or  she  is  responsible  for. 
In  that  capacity,  a  program  manager  is  typically  appointed  to 
each  of  the  contracts  and  is  subordinate  to  the  program 
director.  For  a  source  selection  imder  these  circumstances,  the 
program  director  would  more  than  likely  be  either  the  SSAC 
chairperson  or,  at  the  very  least,  a  member  of  the  SSAC. 
Ultimately,  however,  the  program  manager  is  responsible  for 
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developing  the  acquisition  strategy,  getting  the  requisite 
approvals  to  pursue  the  strategy,  implementing  the 
acquisition  strategy,  and  executing  the  program  after  contract 
award. 

Procuring  Contracting  Officer  (PCO):  The  PCO  is 
responsible  for  all  commimications  with  the  offerors,  and 
ensuring  that  the  entire  source  selection  process  is  consistent 
with  the  FAR  and  its  internal  department-  or  command- 
unique  regulations.  The  PCO  is  also  responsible  for  advising 
the  SSAC  on  the  interpretation  of  the  findings  to  ensure  a 
consistent  and  objective  award  decision.  Some  organizations 
have  had  some  of  their  procurement  personnel  trained  in 
SCE. 

Legal  Advisor  /  Attorney:  All  source  selections  require  some 
degree  of  interaction  with  the  acquisition  organization's  legal 
staff.  For  this  reason  a  JA  individual  is  normally  a  member  of 
the  SSAC.  Some  organizations  have  had  legal  personnel 
trained  in  SCE.  This  training  is  extremely  important  because 
source  selections  are  implemented  many  different  ways 
throughout  the  government  and  acquisition  organizations 
may  employ  different  techniques  even  within  the  same 
government  agency. 

Software  Capability  Evaluation  (SCE)  Team:  This  is  the 
group  of  four  to  six  people  that  will  conduct  the  SCE  site  visits 
for  the  source  selection.  The  SCE  team  could  evaluate  the 
SPIP,  Software  Development  Plan  (SDP),  and  software 
process-related  portions  of  the  proposals.  The  unique 
circumstcmces  of  each  contract  will  dictate  the  extent  of  the 
contribution  the  SCE  team  must  make  to  the  source  selection 
(e.g.  RFP  familiarity,  proposal  familiarity,  and  SSEB 
participation). 

SCE  Team  Leaden  This  individual  will  plan,  prepare,  and 
execute  the  SCE.  The  SCE  team  leader's  responsibilities 
include  advising  the  SSEB/SSET  Chairperson  regarding  the 
specific  findings  of  tire  SCE  team  and  documenting  those 
findings  in  writing;  in  addition  the  team  leader  may  be 
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required  to  brief  the  SCE  portion  of  the  technical  area. 
Personnel  with  previous  SCE  experience  should  be  assigned 
to  this  position. 

Contractor  SCE  Point  of  Contact:  This  is  the  development 
organization's  focal  point  for  an  SCE  site  visit.  The  SCE  team 
leader  will  work  with  this  individual  to  minimize  the  impact 
of  SCE  interviews  and  documentation  gathering  on  the 
evaluated  organization.  The  interaction  will  begin  before  the 
site  visit  to  coordinate  activities  and  request  documentation 
or  organizational  charts.  An  important  point  to  remember  is 
that  all  discussions,  both  plaimed  and  implanned,  after  the 
issuance  of  the  REP  must  be  through  the  PCO. 
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AFSB  Air  Force  Science  Board 

AMC  Army  Materiel  Command 

AMIS  Acquisition  Management  Information  System 

BAFO  Best  and  Final  Offer 

CAO  Contract  Administration  Office 

CBD  Commerce  Business  Daily 

CDR  Critical  Etesign  Review 

CMM  Capability  Maturity  Model 

CPAR  Contractor  Performance  Analysis  Report 

CPEP  Contractor  Performance  Analysis  Program 

CRs  Clarification  Requests 

CSC  Computer  Software  Component 

CSCI  Computer  Software  Configuration  Item 

CSU  Computer  Software  Unit 

DCAA  Defense  Contracting  Audit  Agency 

DoD  Department  of  Defense 

DTIC  Defense  Technical  Information  Center 

Dem/Val  Demonstration /Validation 

DRs  Deficiency  Reports 

DSMC  Defense  Systems  Management  College 

HMD  Engineering  Manufacturing  Development 
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ESC 

FAR 

FCA 

GAO 

IFPP 

ms 

JPO 

JTIDS 

KPA 

KSLOC 

LTR 

MCCR 

MMP/CR 

MQ 

NAWC 

NRAD 

NTE 

PCA 

PCO 

PDR 

PEO 

PFN 


Electronic  Systems  Center  (formally  ESD) 

Federal  Acquisition  Regulation 
Functional  Configuration  Audit 
General  Accoimting  Office 
Instructions  for  Proposal  Preparation 
Interface  Requirements  Specification 
Joint  Program  Office 

Joint  Tactical  Information  Distribution  System 

Key  Process  Area 

Thousand  Source  Lines  of  Code 

Letter 

Mission  Critical  Computer  Resources 

Manufacturing  Management  Production/Capa¬ 
bility  Review 

Maturity  Questionnaire 

Naval  Air  Warfare  Center 

NCCOSC  (Naval  Command,  Control,  and  Ocean 
Surveillance  Center)  RDT&E  (Research  Develop¬ 
ment  Test  and  Engineering)  Division 

Not  to  Exceed 

Physical  Configuration  Audit 
Procuring  Contracting  Officer 
Preliminary  Design  Review 
Program  Executive  Officer 
Point  For  Negotiation 
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PM 

Program  Manager 

PRAG 

Performance  Risk  Analysis  Group 

RAI 

Request  for  Additional  Information 

RFP 

Request  For  Proposal 

SCE 

Software  Capability  Evaluation 

SCM 

Software  Configuration  Management 

SDD 

Software  Design  Document 

SDP 

Software  Development  Plan 

SDIO 

Space  Defense  Initiative  Office 

SDR 

System  Design  Review 

SEI 

Software  Engineering  Institute 

SEPG 

Software  Engineering  Process  Group 

SOW 

Statement  of  Work 

SPIP 

Software  Process  Improvement  Plan 

SPA 

Software  Process  Assessment 

SQA 

Software  Quality  Assurance 

SRR 

System  Requirements  Review 

SRS 

Software  Requirements  Specification 

SSA 

Source  Selection  Authority 

SSAC 

Source  Selection  Advisory  Council 

SSDD 

System /Segment  Design  Document 

SSEB 

Source  Selection  Evaluation  Board 

SSET 

Source  Selection  Evaluation  Team 

SSEG 

Source  Selection  Evaluation  Guide 
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SSP  Source  Selection  Plan 

USAF  United  States  Air  Force 


B-120 


CMU/SEI-94-TR-5 


Appendix  C:  Bibliography 


Appendix  C 

Bibliography 

[ASD  87] 

Aeronautical  Systems  EHvision  (ASD),  Acquisition  Management 
Pamphlet  800-5,  "Software  Development  Capability/Capacity 
Review."  Department  of  the  Air  Force,  Air  Force  Systems 
Commeuid  (AFSC),  10  September  1987. 

[AFSC  90] 

Air  Force  Systems  Command,  AFSC  Pamphlet  800-51  "Software 
Development  Capability  Assessment  (SDCA),"  November  1990. 

[AFSB  89] 

Air  Force  Studies  Board  (AFSB)  Report,  "Adapting  Software 
Development  Policies  to  Modem  Technology."  Commission  on 
Engineering  eind  Technical  Systems,  National  Research  Coxmcil, 
National  Academy  Press,  1989. 

[Bender  87] 

Bender,  David.  Computer  Law.  New  York;  Matthew  Bender,  1987. 

[Bigelow  87] 

Bigelow,  Robert  P.  Computer  Contracts.  New  York:  Matthew  Bender, 
1987. 

[Boehm  91] 

Boehm,  B.W.  "Software  Risk  Management:  Principles  and 
Practices,"  IEEE  Software  8, 1  0anuary  1991):  32-41. 

[Bucholz  87] 

Bucholz,  S.,  cmd  Roth,  T.  Creating  the  High  Performance  Team.  New 
York:  Wiley,  1987. 

[Carlucd  81] 

Carlucd,  F.  C.,  HI,  "Improving  the  Acquisition  Process." 
Memorandum  for  Secretaries  for  the  Military  Departments,  et  al. 
Washington,  D.C.,  April  30, 1981. 

[Crosby  88] 

Crosby,  P.B.  Quality  is  Free.  McGraw-Hill,  New  York,  NY,  1979. 

[Dart  87] 

Dart,  S.;  and  Ellison,  B.  Software  Development  Environments  (CMU/ 
SEI-87-TR-24,  ADA  200542).  Pittsburgh,  Pa.:  Software  Engineering 
Institute,  Carnegie  Mellon  University,  1987. 

[Deep] 

Deep,  Ron,  et  al.  DSMC  Risk  Marwgement  Guide.  Etefense  Systems 
Management  College,  The  Pentagon,  Washington,  D.C. 

[Deming  86] 

Deming,  W.  Edwards.  Out  of  the  Crisis,  MIT  Center  for  Advanced 
Engineering  Study,  Cambridge,  Ma,  1986. 

[Denton  89] 

Denton,  D.  Keith.  "Four  Steps  to  Resolving  Conflicts."  Quality 
Progress  22, 4  (April  1989):  29-33. 

CMU/SE1-94-TR-5 


C-121 


Appendix  C:  Bibliography 


ID0D88] 

[FAR] 

[GAO  86] 
[Humphrey  89] 
[Humphrey  88] 
[Humphrey  87a] 

[Humphrey  87b] 

Quran  88] 

[Kelly  91] 

[Klein  87] 

[Paulk  93a] 

[Paulk  93b] 


Department  of  Defense  Standard  2167A  (DoD-STD-2167A), 
"Defense  System  Software  Development."  29  February  1988, 
superseding  Dol>STD-2167. 

Federal  Acquisition  Regulation,  FAR  Supplement  242.70,  "Contract 
Administration."  FAR  15.6,  "Source  Selection."  FAR  52.215-1, 
"Ex<imination  of  Records  by  Comptroller  General." 

Technical  Risk  Assessment — ^The  Status  of  Current  DoD  Efforts, 
General  Accounting  Office,  April  1986. 

Humphrey,  Watts  S.  Managing  the  Software  Process.  Reading  Ma.: 
Addison-Wesley,  1989. 

Humphrey,  Watts  S.  "Characterizing  the  Software  Process;  A 
Maturity  Framework."  IEEE  Software  5, 2  (March  1988):  73-79. 

Humphrey,  Watts.  Characterizing  the  Software  Process:  A  Maturity 
Framework  (CMU/SEI-87-TR-11,  ADA1828952).  Pittsburgh,  Pa.: 
Software  Engineering  Institute,  Carnegie  Mellon  University,  1987. 

Humphrey,  W.;  and  Sweet,  W.  A  Method  for  Assessing  the  Software 
Engineering  Capability  of  Contractors  (CMU/SEI-87-TR-23, 
ADA187230).  Pittsburgh,  Pa.:  Software  Engineering  Institute, 
Carnegie  Mellon  University,  1987. 

Jurein,  J.M.  Juran  on  Planning  for  Quality.  New  York:  Macmillan, 
1988. 

Kelly,  Mark.  The  Adventures  of  a  Self-Managing  Team.  San  Diego: 
Pfeiffer  &  Co.,  1991. 

Klein,  D.;  &  Firth,  R.  Final  Evaluation  of  MIPS  M/500  Final  Report  of 
the  RISC  Insertion  Project  (CMU/SEI-87-TR-25,  ADA  200611). 
Pittsburgh,  Pa.:  Software  Engineering  Institute,  Carnegie  Mellon 
University,  1987. 

Paulk,  Mark  C.,  et.al.  Capability  Maturity  Model  For  Software,  Version 
1.1  (CMU/SEI-93-TR-24).  Pittsburgh,  Pa.:  Software  Engineering 
Institute,  Carnegie  Mellon  University,  1993. 

Paulk,  Mark  C.,et.al.  Key  Practices  of  the  Capability  Maturity  Model 
(CMU/SEI-93-TR-25).  Pittsburgh,  Pa.:  Software  Engineering 
Institute,  Carnegie  Mellon  University,  1993. 


C-122 


CMU/SEI-94-TR-5 


Appendix  C:  Bibliography 


[SCE  93] 


[USAF  88] 


Software  Capability  Evaluation  Project.  Software  Capabiblity 
Evluation  Version  2.0  Method  Description  (CMU/SEI-93-TR-17). 
Pittsburgh,  Pa.:  Software  Engineering  Institute,  Carnegie  Mellon 
University,  1993. 

United  States  Air  Force.  Air  Force  Regulation  70-15,  Air  Force 
Federal  Acquisition  Regulation  Supplement,  1990. 


CMU/SEI-94.TR-5 


C-123 


Aii^ndix  C:  Bibliography 


C-124 


CMU/SEI-94-TR-5 


Appendix  D;  Detailed  SCE  Implenientation  Checklist 


Appendix  D  SCE  Implementation  Checklist 


The  following  checklist  has  been  extracted  from  the  text  of  the 
document. 


Acquisition  Start 


□  Develop  initial  awareness  (SPO  and  PCO  organizations) 

□  Determine  applicability  to  this  acquisition 

□  Review  existing  SCE  policies  and  procedures 

□  Review  acquisition  strategy 

□  Determine  SCE  needs  fpr  acquisition 

□  Develop  SCE  implementation  recommendation 

□  Input  to  acquisition  strategy  document 

□  Champion  implementation  of  SCE  on  acquisition 

□  Obtain  commitment  to  use  SCE 


Organize /Select 
SCE  Team 


□  Review  SCE  team  leader  and  team  member  qualification 
criteria 

□  Ensure  appropriate  criteria  for  team  are  applied  to 
acquisition 

□  Attend  SCE  Overview  (for  appropriate  acquisition 
personnel) 

□  Prepare  candidate  SCE  team  member  list 

□  Obtain  commitment  from  candidate  team  member's 
organization 

□  Familiarize  team  with  acquisition  policy 

□  Attend  SCE  training 
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Execute 

Acquisition  Start 
Phase 


□  Determine  SCE  placement  within  source  selection 
documentation 

□  Prepare  recommendation  on  how  SCE  findings  will  be 
integrated  into  the  acquisition 

□  Develop  Product  Profile 

□  Determine  Target  Process  Capability  (TPC) 

□  Determine  disposition  of  SCE  data 

□  Estimate  number  of  contractor  sites  to  be  visited 

□  Estimate  resources  and  time  (manpower,  travel,  support) 

□  Determine/schedule/implement  preliminary  SCE  tasks 

□  Complete  CBD  announcement  input 

□  Prepare  Pre-proposal  Conference  Briefing 

□  Insert  Acquisition  Plan,  Source  Selection  Plan,  RFP  SCE 
language 

□  Request  completion  of  Mahuity  Questionnaire  and  project 
profiles 

□  Instructions  on  how  to  submit  material 

□  Prepare  Evaluation  Standards 


Execute  General 
and  Specific 
Preparation 
Phases 


□  Schedule  SCE  team  to  meet  and  execute  SCE  Method 
pre-site  visit  preparation. 

□  Analyze  project  profiles 

□  Select  contractor  projects 

□  Identify  critical  subprocess  for  all  contractors 

□  Determine  key  issues  for  individual  contractors 

□  Develop  initial  exploratory  interview  questions  and 
identify  initial  set  of  documents  for  review 

□  Develop  and  notify  contractor  points  of  contact  regarding 
SCE  team  logistical  requirements  (10  working  days  in 
advance) 

□  Arrange  site  logistics  (room,  table,  chairs,  documents, 
preliminary  on-site  and  interview  schedules,  computing 
needs,  etc.) 
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Conduct  SCE 


□  Conduct  SCE  site  data  collection 

□  Conduct  in-briefing  with  on-site  contractor 

□  Analyze  organizational  and  project  documentation 

□  Review  and  modify  agenda  and  schedule  as  necessary 

□  Conduct  exploratory  interviews 

□  Request  additional  documentation 

□  Validate  interview  responses 

□  Draft  preliminary  findings 

□  Validate  preliminary  findings 

□  Conduct  consolidation  interviews 

□  Validate  improvement  activities 

□  Develop  final  findings 

□  Conduct  exit  briefing  (as  prescribed  by  Procuring 
Contracting  Officer  (PCO) 


Write  /  Submit 
Final  Report  to 
Acquisition 
Organization 


□  Document  conduct  of  SCE  and  rationale  for  findings 

□  Document  effort  and  resources  expended 

□  Develop  lessons  learned  and  provide  feedback  to  improve 
SCE  Method 


Assist 
Acquisition 
Organization’s 
Use  of  SCE 
Findings 


□  Develop  and  deliver  final  SCE  results  briefing  to  SSEB  (if 
necessary) 

□  Consult  with  ^EB  and  SSAC  as  needed  (elaborate  on  SCE 
findings) 

□  Assist  SSEB  in  preparing  and  delivering  formal  SCE 
presentation  to  the  SSAC 


CMU/SEI-94-TR-5 


D-127 


Appendix  0:  Detailed  SCE  Implennentation  Checklist 


Formal 

Feedback 


□  Conduct  SCE  findings  briefing  for  winning  contractor 

□  Conduct  SCE  findings  for  unsuccessful  offerors 

□  Dispose  of  SCE  data  (in  accordance  with  acquisition 
guidelines) 

□  Disband  SCE  team 
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Appendix  E  Templates 

The  figures  in  this  appendix  have  been  extracted  from 
Chapter  5  of  this  document.  The  figures  show  the  kind  of 
information  about  SCE  that  should  be  included  in  acquisition 
documents. 


As  part  of  <Acquisition  Agency’s>  overall  effort  to  improve  software  quality,  cost,  and  schedule 
performance,  the  software  process  capability  of  the  responding  offerors  will  be  a  consideration  in  the 
source  selection  decision.  <Acqui8ition  Agency>  will  use  the  Software  Capability  Evaluation  (SCE) 
method  developed  by  the  Software  Engineering  Institute  (SEI)  to  evaluate  the  current  software 
process  of  responding  offerors  (see  Capability  Maturity  Model  for  Software  (CMU/SEt-93-TR-24, 
February  1993).  Offerors  who  are  determined  to  be  in  the  competitive  range  will  have  their  software 
process  capability  verified  by  an  SCE  team.  The  SCE  team  will  analyze  key  process  areas  (KPAs) 
through  the  Defined  maturity  level  and  look  for  a  software  process  improvement  program  leading  to 
levels  of  process  capability  higher  than  the  current  capability.  A  conference  will  be  held  at  <time>  on 
<date>  at  <where>  to  answer  any  questions. 


Figure  E*1  Sample  CBD  Announcement 


3.0  Source  Screening 

3.1  Screening  Criteria.  Initial  screening  of  potential  offerors  was  conducted  by  the  publication  of  a 
sources-sought  synopsis  in  the  Commerce  Business  Daily  on  <date>.  The  synopsis  requested 
that  interested  offerors  provide  their  qualifications  in  the  following  areas: 

a.  Software  Engineering  Capability.  Knowledge  of  software  process  improvement  with  a 
verifiable  program  for  process  improvement  using  the  Software  Capability  Evaluation  (SCE) 
Method  devebped  by  the  Software  Engineering  Institute  (SEI)  to  evaluate  the  current  software 
process  of  responding  offerors  (Capability  Maturity  Model  (CMM)  (CMU/SEI-93-TR-24,  Feb 
93))  The  offerors  who  are  determined  to  be  in  the  competitive  range  after  initial  government 
evaluation  of  proposals  is  completed  will  have  their  software  process  capability  verified  by  an 
SCE  team.  The  SCE  team  will  evaluate  key  process  areas  (KPAs)  through  the  Defined 
maturity  level  and  look  for  a  demonstratable  software  process  improvement  program  leading 
to  levels  of  process  capability  higher  than  the  current  capability.  Do  not  provide  Software 
Process  Assessment  (SPA)  results. 


Figure  E*2  SCE  as  a  Screening  Criterion  in  the  SSP 
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6.4  Evaluation  Criteria 


6.4.1  Assessment  Criteria 

a  Soundness  of  approach 

b.  Compliance  with  requirements 

6.4.2  Specific  Criteria 

a.  Technical  Area  -  This  area  will  evaluate  three  items  (Software  Engineering  Capability  (SEC), 
Technical  (TECH)  approach,  and  Management)  represented  here  in  descending  order  of 
importance: 

1 .  Software  Engineering  Capability.  The  government  will  evaluate  the  software  process  by 
reviewing  the  offeror’s  Software  Process  Improvement  Plan  (SPIP)  and  by  using  the 
Software  Engineering  Institute-  (SEI)  developed  Software  Capability  Evaluation  (SCE) 
Method.  The  government  will  determine  the  software  process  capability  by  investigating  the 
Key  Process  Area  (KPAs)  defined  in  the  SEI  Technical  Report,  Cape^lity  Maturity  Model  for 
Software  (CMU/SEI-93-TR-24,  February  1993).  The  report  contains  a  description  of  the 
Capability  Maturity  Model  (CMM).  The  government  will  perform  an  SCE  of  each  offeror 
remaining  in  the  competitive  range  by  reviewing  current  projects  at  the  site  proposirrg  on  this 
contract  and  comparing  processes  used  on  those  prefects  in  the  written  proposal/SPIP. 

The  evaluation  will  result  in  a  composite,  substantiated  through  individual  interviews  and 
reviews  of  documentation,  of  the  offeror’s  software  process  on  ttie  government-selected 
projects.  A  risk  assessment  to  compare  proposed  practices  to  current,  validated  practices 
may  be  performed.  The  evaluation  team  will  determine  findings  of  the  offeror’s  strengths, 
weaknesses,  and  improvement  activities  in  all  KPAs  through  the  Defined  maturity  level. 
Results  of  the  SCE  will  not  be  pass/fail.  Identified  weaknesses  will  be  provided  in  writing 
during  subsequent  discussions.  The  offeror  will  be  allowed  to  respond  to  the  findings  with  a 
limited  number  of  pages  of  text.  The  on-site  evaluators  may  be  separate  and  distinct  from  the 
proposal  evaluation  team  and  may  include  a  government  contracting  representative.  Alt 
evaluators  have  been  trained  in  the  SCE  Method. 


Figure  E-3  SCE  as  a  Specific  Criterion  For  The  SSP 


E-130 


CMU/SEI-94-TR-5 


Appendix  E;  Templates 


6-3  Present  and  past  perfomuince  (Parformanca  risk) 

(1 )  Present  and  past  performance  is  a  consideration  in  all  acquisition  agency  source 

seiections.  Performance  risk  is  a  structured  treatment  of  present  and  past  performance 
and  will  be  determined  for  each  area.  It  is  a  confidence  measure  that  assesses  the 
offeror's  present  and  past  work  record  in  order  to  determine  the  offeror’s  ability  to  perform 
the  proposed  effort.  Performance  risk  is  assessed  by  the  Performance  Risk  Analysis  Group 
(PRAG),  which  should  be  chaired  by  a  program  manager-level  (senior)  individual  and 
consist  of  government  personnei.  Performanr^  evaluation  and  risk  assessment  focus  on 
relevant  past  performance  as  it  relates  to  the  specific  criteria.  It  is  the  PRAG’s 
responsibility  to  analyze  the  data  collected;  determine  its  relevance;  and  to  perform  an 
“independent*  risk  assessment.  The  results  of  the  SCE  will  also  be  factored  into  the  overall 
performance  risk  rating  assigned  by  the  PRAG.  For  information  on  how  to  establish  a 
PRAG,  how  it  operates,  the  forms  which  are  used,  and  how  the  evaluation  is  made  and 
reported,  refer  to  the  acquisition  agency  Source  Selection  Handbook. 


Figure  E-4  SCE  as  part  of  the  PRAG  for  SSP 


AGENDA 

•• 

SCE  description 

USE  OF 

SCE  team  activities 

SOFTWARE  CAPABILITY  EVALUATION 

Key  points  about  the  site  visit 

(SCE) 

Criteria  for  validating  practices 

IN  THE 

Sample  of  documents  reviewed 

XYZ  CONTRACT 

1 

Example  of  typical  site  visit  schedule 

Key  Process  Areas  (KPAs)  investigated 

Example  detailed  summary  findings 

Figure  E-5  Pre-proposai  Conference  Title  and  Agenda 
Slides 
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SAMPLE  OF  DOCUMENTS  REVIEWED 

•  Current  organization  chart. 

•  Corporate  policies/procedures  on  software 
management. 

•  Project  policies/procedure  on  software 
management. 

•  Software  development  plans. 

•  Software  configuration  management  plans. 

•  Software  quality  assurance  plans. 

•  Training  plans. 

•  Current  metrics  (timing  and  sizing,  etc.). 


•  Implementation  procedures  and  company 
standards. 


Figure  E-7  Pre-proposal  Conference  Documentation  Slide 
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SAMPLE  SITE  VISIT  SCHEDULE 

DAYO 

0800-1330  Travel  for  SCE  team 
1330-1800  Prepare  for  site  visit 

DAY1 

0800-0830  Contractor  welcome/introduciion  and  SCE  team’s  orientation  briefing 
0830- 1 030  Contractor  software  presentation/contractor  understanding  caucus 
1030-1230  Initial  review  of  organizational  documents 
1230-1330  Lunch 

1330-1430  SCE  team  interviews  -  senior  managers 
1 430- 1 600  SCE  team  intennews  -  program  managers 
1 600-1 730  SCE  team  interviews  -  software  managers 
1 730-1 800  SCE  team  caucus  and  requests  documents 

DAY  2 

0800-1 000  SCE  team  interviews  continued  with  program  managers,  software 
managers 

1000-1200  SCE  team  inten/iews  •  managers  (engineering,  SQA,  SCM,  test,  SEPG) 
1200-1300  Lunch 

1300-1 400  SCE  team  caucus,  request  and  review  documents 


DAY  3 

0800-1000  SCE  team  interviews  •  key  personnel  (may  include  engineering  staff) 
1000-1200  Review  additional  documentation 
1200-1300  Lunch 

1300-1500  Additional  documentation  review  and/or  additional  SCE  team  interviews  ■ 
Consolidation  interviews  with  managers,  engineers,  and  staff 
1 500- 1 800  SCE  team  caucus  and  preparation  of  findings 


DAY  4 

0800-0900 

0900-1000 

1000-1100 

1100-1730 


Final  preparation  of  findings 
Exit  meeting  with  offeror 
SCE  team  caucus  and  wrap-up 
Travel  for  SCE  team 


Figure  E-8  Pre>proposal  Conference  Site  Visit  Scheduie 
Siide 
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Level 

Characteristic 

Key  Process  Area  L 

Optimizing  (5) 

•  Continuous  process 
improvement  capability 

•  Process  change  management 

•  Technology  innovation 

•  Defect  prevention 

Managed  (4) 

•  Product  quality  planning 
and  tracking  of 
measured  software  pro¬ 
cess 

•  Process  measurement  and 
analysis 

•  Quality  management 

Defined  (3) 

•  Life  cycle  process 
defined  and  institutional¬ 
ized  to  provide  product 
quality  control 

•  Peer  Reviews 

•  Intergroup  coordination  I 

•  Software  product  engineering  | 

•  Integrated  software  1 

management  | 

•  Training  program  I 

•  Organization  process  1 

definition  I 

•  Organization  process  focus  I 

Repeatable  (2) 

•  Management  oversight 
and  tracking  of  project 

•  Stable  plannmg 

•  Software  configuration  I 

management 

•  Software  quality  assurance 

•  Software  subcontract 

management  | 

•  Softwar?'  project  tracking  and 

oversight  I 

•  Software  project  planning  I 

•  Requirements  management  I 

Initial  (1) 

•  Ad  hoc  (unpredictable, 
chaotic) 

1 

Figure  E-9  Pre-proposal  Conference  CMM  Slide 
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EXAMPLE  SUMMARY  FINDING 
Strong 

•  Software  Quality  Assurance  (SQA) 

Araeptable 

•  Software  Project  Planning 

•  Software  Configuration  Management 

•  Requirement  Management 

Weak 

•  Software  Subcontract  Management 

•  Organization  Process  Definition 

•  Training  Program 

•  Peer  Reviews 


SAMPLE  DETAILED  HNDING 
SQA  KPA 

Strengths 

•  independent  reporting  chain 

•  Highiy  visibie 

•  Insuring  software  engineering  standards 
compliance 

•  Management  commitment  -  strong  staff 

Weaknesses 

•  Inconsistent  auditing 

•  Ineffective  use  of  resources 

Improvement  Activities 

•  Establishing  procedures  for  consistent 
auditing 


Figure  E<10  Pre-Proposal  Conference  Sample  Findings 
Slides 
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B.  Evaluation  Criteria 

1 .0  Introduction 

2.0  Basis  for  Contract  Award 

3.0  General  Considerations 

4.0  Assessment  Criteria 

4.1  Specific  Criteria 

4.1.1  Technical  Area 

a.  Technical  Area  -  This  area  will  evaluate  three  items  (SEC,  TECH  approach  and 
Management)  represented  here  in  descending  order  of  importeince: 

1 .  Software  Engineering  Capability.  The  government  wili  evaluate  the  software  process  by 
reviewing  the  offeror's  Software  Process  Improvement  Plan  (SPIP)  and  by  using  the 
Software  Engineering  institute  (SEI)-deveioped  Software  Capability  Evaluation  (SCE).  The 
government  will  determine  the  software  process  capability  by  investigating  the  Key 
Process  Area  (KPAs)  defined  in  the  SEI  Technical  Report,  Capability  Maturity  Model  for 
5oflfw8r&(CMU/SEI-93-TR-24).  The  report  contains  a  description  of  the  Capability  Maturity 
Model  (CMM)  and  the  SEI-defined  maturity  levels.  The  government  will  pertonn  an  SCE  of 
each  offeror  remaining  in  the  competitive  range  by  reviewing  current  projects  at  the  site 
proposing  on  this  contract  and  comparing  methods/processes  used  on  those  projects  in 
the  written  proposal/SPIP. 

The  evaluation  wiil  result  in  an  organizational  composite,  substantiated  through  individual 
intenriews  and  reviews  of  documentation,  of  the  offeror’s  software  process  activities  on  the 
government-selected  projects.  A  risk  assessment  to  compare  proposed  practices  to 
current,  validated  practices  may  be  perfonned.  The  evaluation  team  will  determine  findings 
of  the  offeror’s  strengths,  weaknesses,  and  improvement  activities  in  all  KPAs  through  the 
Defined  maturity  level.  Results  of  the  SCE  will  not  be  pass/fail.  Identified  weaknesses  virill 
be  provided  in  writing  during  subsequent  discussions.  The  offeror  will  be  allowed  to 
respond  to  the  findings  with  a  limited  number  of  pages  of  text.  The  on-site  evaluators  may 
be  separate  and  distinct  from  the  proposal  evaluation  team  and  may  include  a  government 
contracting  representative.  All  evaluators  have  been  trained  in  the  SCE  Method. 

4.2  Cost/Price  Area... 


Figure  E-1 1  General  RFP  Language  To  Include  SCE  (RFP 
Section  M) 


3.1  General  Instructions  The  Technical  Proposal  shall  consist  of  the  Executive  Summary  and 
<n>  sections... 

3.2  Executive  Summary... 


3.3  Specific  Instructions... 

3.3.1  Section  I  -  Software  Engineering  Capability.  The  offeror  will  provide  the  following 

information  to  assist  the  government’s  preparation  for  the  Software  Capability  Evaluation  of 
each  offeror 

a.  The  offeror  wilt  complete  the  SEI  Maturity  Questionnaire  (MQ)  (current  version)  using  the 
Assessment  Recording  Form  for  eight  current  projects  at  the  site  proposing  on  this 
solicitation  (a  project  is  acceptable  only  if  it  has  been  completed  in  the  last  year).  The 
offeror  should  select  those  projects  that  best  match  the  engineering  requirements  of  this 
contract.  For  offerors  with  fewer  than  eight  current  projects  at  the  proposing  site,  submit 
MQ  responses  for  as  many  projects  as  are  available.  For  each  ‘Ves*  response,  please 
note  on  the  comment  line  the  mechanism  or  documentation  justifying  the  response.  The 
MQ  can  be  found  in  Atch  XXX  of  the  IFPP.  llte  completed  Assessment  Recording  Forms 
will  be  submitted  with  the  proposal.  For  all  responses,  please  note  at  the  start  of  the 
comment  line  the  degree  of  implementation  of  each  practice  using  a  letter  identifier  from 
the  following  legend: 

A  •  Not  implemented  at  this  time. 

B  •  Not  implemented  at  this  time,  but  desired. 

C  -  Currently  planning  to  implement.  See  improvement  plan. 

D  -  In  the  process  of  implementing. 

E  •  Implemented  with  less  than  a  year’s  experience. 

F  -  Implemented  on  a  project-by-project  basis. 

G  -  Implemented  organizationally. 

H  -  Not  appropriate  for  our  organization. 


Figure  E;'12  Instructions  For  Completing  Assessment  Re¬ 
cording  Forms 
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linuec 


b.  For  each  project,  the  offeror  will  complete  a  Project  Profile,  attach  it  to  the  respective 
Assessment  Recording  Form,  and  submit  it  with  the  proposal.  The  Project  Profile  template 
can  also  be  found  in  Atch  XXX  of  the  IFPP.  This  document  shall  be  no  greater 
than  one  page  per  project. 


Figure  E-13  Instructions  For  Completing  Project  Profiles 


3.31  (continued) 


c.  The  offeror  will  submit  the  proposing  site’s  Software  Process  Improvement  Plan  (SPIP), 
in  the  format  of  their  choosing,  with  the  proposal.  The  document  will  be  no  more  than  15 
pages.  The  SPIP  should  communicate  the  offeror’s  current  software  process  capability  as 
well  as  their  desired  maturity  level,  specific  planned  improvements,  dedicated  resources, 
effort  estimates,  and  a  time  phasing  of  those  improvements  to  bring  the  offeror’s  software 
process  capability  to  the  organization's  desired  maturity  level. 


Figure  E-14  Instructions  For  Submitting  the  Software 
Process  Improvement  Plan  (SPIP) 
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d.  After  the  proposal  is  received,  the  government  will  coordinate  a  site  visit  with  those 
offerors  remaining  in  the  competitive  range  to  conduct  the  Software  Capability  Evaluation 
(SCE)  at  the  offeror’s  location.  The  offeror  wilt  provide,  with  your  proposal,  a  point  of 
contact  and  phone  number  at  the  offeror’s  site  for  the  SCE  team  leader  to  coordinate  all 
SCE  activities.  The  government  will  also  communicate  details  about  the  site  visit  during 
the  coordination  process.  The  offeror  will  be  notified  of  the  projects  to  be  examined 
approximately  five  working  days  prior  to  the  site  visit.  The  site  visit  dates  selected  by  the 
government  are  not  open  for  discussion. 

e.  If  a  site  visit  is  conducted  with  your  firm,  the  SCE  team  will  need  a  closed  meeting  room 
capable  of  accommodating  at  least  eight  people.  The  offeror  should  have  a  copy  of  the 
organization’s  software  standards,  procedures  and/or  operating  instructions,  and 
organizational  charts  for  the  projects  being  reviewed  in  the  meeting  room  when  the  SCE 
team  arrives.  All  interviews  conducted  as  part  of  the  SCE  will  be  done  in  private,  one 
individual  at  a  time. 

f.  The  Assessment  Recording  Forms,  Project  Profiles,  and  Software  Process  Improvement 
Plans  will  not  be  included  in  the  page  count  limitations  for  the  proposal. 


Figure  E*15  Instructions  For  Site  Visit  Coordination 


E-140 


CMU/SEI-94-TR-5 


Appendix  E;  Templates 


DESCRIPTION:  Software  Engineering  Capability— The  government  will  evaluate  the  offeror’s 
organizational  software  process  capability  by: 

•  Performing  a  Software  Capability  Evaluation  (SCE). 

•  Evaluating  the  offeror's  program  for  software  process  improvement. 

•  Evaluating  the  extent  to  which  the  offeror’s  software  process  supports  the  goals,  objectives,  and  require¬ 
ments  of  the  solicitation. 

STANDARD-  The  standard  Is  met  when  the  offeror  presents  a  sound,  compliant  approach  and: 

1 .  The  findings  from  the  SCE  show  that  the  offeror  is  strong  or  acceptable  in  each  of  the  following  key 
process  areas: 

•  Software  Configuration  Management 

•  Software  Quality  Assurance 

•  Software  Subcontract  management 

•  Software  project  tracking  and  oversight 

•  Software  project  planning 

•  Requirements  Management 

2.  The  findings  from  the  SCE  show  that  the  offeror  is  strong  or  acceptable  in  at  least  one  of  the  following 
software  process  areas: 

•  Peer  Reviews 

•  Intergroup  coordination 

•  Software  product  engineering 

•  Integrated  software  management 

•  Training  program 

•  Organization  process  definition 

•  Organization  process  focus 

3.  The  Software  Process  Improvement  Plan  submitted  with  the  offeror’s  proposal  portrays  the  offeror’s 

current  process  capability  realistically  and  presents  a  realistic  plan  for  process  improvement.  The 
offeror’s  plan  is  consistent  with  the  SCE  findings.  The  SPIP  outlines  the  offeror’s  plan  to  achieve 
higher  maturity  levels  and  demonstrates  that  the  offeror  understands  software  process  improvement, 
both  technically  and  in  the  effort  required  to  increase  and  sustain  higher  maturity  levels.  I 


Figure  E-16  Streamlined  SCE  Evaluation  Standard 
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Appendix  F  Pre-Proposal  Conference  Slides 


The  purpose  of  the  pre-proposal  conference  is  to  explain  the 
SCE  process  and  solicit  feedback  from  the  prospective 
offerors.  The  information  on  the  slides  in  this  appendix  is 
identical  to  the  information  in  the  examples  shown  in  Chapter 
5  of  slides  that  may  be  used  in  a  pre-proposal  conference  (see 
pages  5-91  to  5-99). 

In  this  appendix  the  slides  have  been  enlarged  so  that  they 
may  be  used  directly  in  a  presentation. 
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Use  of  Software 
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SCE  Description 

A  method  for  evaluating  the  software  process  of  an 
organization  to  gain  insight  into  its  software 
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Sample  of  Documents  Reviewed 
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Sample  Detailed  Finding:  SQA  KPA 

Strengths 

•  Independent  reporting  chain 
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Improvement  Activities 

•  Establishing  procedures  for  consistent  auditing 


Example  Summary  Finding 

strong 

•  Software  Quality  Assurance  (SQA) 
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